[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:05 dist-test-slave-r706 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:05 dist-test-slave-r706 krb5kdc[395](info): set up 2 sockets
Jun 20 02:28:05 dist-test-slave-r706 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 02:28:06.678825 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.011s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:06 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:06.869222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39791
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:07.386101 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:07.386546 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:07.386762 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:07.396009 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:07.396245 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:07.396385 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:07.396541 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:07.396663 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:07.408949 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:07.410743 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:07.412578 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:07.429612 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:07.429664 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:07.429541 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:07.431571 408 server_base.cc:1034] running on GCE node
I20260620 02:28:07.432600 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:07.434556 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:07.435827 408 hybrid_clock.cc:648] HybridClock initialized: now 1781922487435725 us; error 66 us; skew 500 ppm
I20260620 02:28:07.436326 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:07 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:07.441061 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:07.443727 408 webserver.cc:460] Webserver started at http://127.0.97.62:42153/ using document root <none> and password file <none>
I20260620 02:28:07.444867 408 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:07.445168 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:07.445566 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:07.449767 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "9b7c12e136334031a516cdbd01b354a4"
format_stamp: "Formatted at 2026-06-20 02:28:07 on dist-test-slave-r706"
server_key: "e941ae4526e0aa1689635ed4baaca928"
server_key_iv: "c9d04c84f509b0f3212f8ba6f201aa9f"
server_key_version: "encryptionkey@0"
I20260620 02:28:07.450796 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b7c12e136334031a516cdbd01b354a4"
format_stamp: "Formatted at 2026-06-20 02:28:07 on dist-test-slave-r706"
server_key: "e941ae4526e0aa1689635ed4baaca928"
server_key_iv: "c9d04c84f509b0f3212f8ba6f201aa9f"
server_key_version: "encryptionkey@0"
I20260620 02:28:07.458200 408 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260620 02:28:07.463140 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:07.464937 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260620 02:28:07.465268 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "9b7c12e136334031a516cdbd01b354a4"
format_stamp: "Formatted at 2026-06-20 02:28:07 on dist-test-slave-r706"
server_key: "e941ae4526e0aa1689635ed4baaca928"
server_key_iv: "c9d04c84f509b0f3212f8ba6f201aa9f"
server_key_version: "encryptionkey@0"
I20260620 02:28:07.465595 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:07.498206 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:07.504029 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:07.504447 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:07.541076 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39791
I20260620 02:28:07.541189 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39791 every 8 connection(s)
I20260620 02:28:07.542888 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:07.546249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 408
I20260620 02:28:07.546664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:07.549696 388 external_mini_cluster.cc:1428] Setting key c36b846f0cca803ca34974fe90868302
I20260620 02:28:07.550052 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:07.573024 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4: Bootstrap starting.
I20260620 02:28:07.578743 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:07.581102 475 log.cc:824] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:07.585840 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4: No bootstrap required, opened a new log
Jun 20 02:28:07 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:07.600272 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c12e136334031a516cdbd01b354a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260620 02:28:07.601125 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:07.601307 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7c12e136334031a516cdbd01b354a4, State: Initialized, Role: FOLLOWER
I20260620 02:28:07.602190 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c12e136334031a516cdbd01b354a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260620 02:28:07.602674 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:07.602967 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:07.603268 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:07.610895 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:07.555602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.0.97.62:39791)
0620 02:28:07.557246 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:07.557265 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:07.559753 (+ 2488us) server_negotiation.cc:407] Connection header received
0620 02:28:07.576956 (+ 17203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:07.576983 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:07.577586 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:07.578067 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:07.583840 (+ 5773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:07.585570 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:07.588814 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:07.589833 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:07.590025 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:07.599667 (+ 9642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:07.599683 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:07.599705 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:07.599767 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:07.604620 (+ 4853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:07.605764 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:07.605772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:07.605778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:07.606040 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:07.606688 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:07.606696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:07.606703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:07.607371 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:07.607796 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:07.608825 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 02:28:07.609382 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":903,"threads_started":1}
I20260620 02:28:07.638041 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c12e136334031a516cdbd01b354a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260620 02:28:07.638960 475 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7c12e136334031a516cdbd01b354a4; no voters:
I20260620 02:28:07.640636 475 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:07.640808 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:07.642267 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9b7c12e136334031a516cdbd01b354a4, State: Running, Role: LEADER
I20260620 02:28:07.643092 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c12e136334031a516cdbd01b354a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260620 02:28:07.644470 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:07.648066 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b7c12e136334031a516cdbd01b354a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c12e136334031a516cdbd01b354a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } } }
I20260620 02:28:07.649371 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:07.649442 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7c12e136334031a516cdbd01b354a4. Latest consensus state: current_term: 1 leader_uuid: "9b7c12e136334031a516cdbd01b354a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c12e136334031a516cdbd01b354a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } } }
I20260620 02:28:07.649961 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:07.654742 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:07.664168 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:07.678464 487 catalog_manager.cc:1347] Generated new cluster ID: bd6f2db1636a44e78395fd8d57301c8d
I20260620 02:28:07.678717 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:07.692138 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:07.693886 487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:07.704141 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b7c12e136334031a516cdbd01b354a4: Generated new TSK 0
I20260620 02:28:07.705227 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:07.761688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39791
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:08.905815 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:08.906186 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:08.906446 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:08.915732 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:08.915971 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:08.916220 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:08.930130 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:08.931653 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:08.933351 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:08.951151 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:08.951216 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:08.951375 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:08.958050 501 server_base.cc:1034] running on GCE node
I20260620 02:28:08.959196 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:08.960563 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:08.961805 501 hybrid_clock.cc:648] HybridClock initialized: now 1781922488961719 us; error 57 us; skew 500 ppm
I20260620 02:28:08.962234 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:08 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:08.966651 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:08.968641 501 webserver.cc:460] Webserver started at http://127.0.97.1:34883/ using document root <none> and password file <none>
I20260620 02:28:08.969597 501 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:08.969823 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:08.970268 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:08.973922 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "75b22fa3e6e2462f8979b45ff413b2ae"
format_stamp: "Formatted at 2026-06-20 02:28:08 on dist-test-slave-r706"
server_key: "0bd15b5c73d78421f6cf90fd44b45a4a"
server_key_iv: "e15c7be78753d2e11d824aa1b575a042"
server_key_version: "encryptionkey@0"
I20260620 02:28:08.974997 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "75b22fa3e6e2462f8979b45ff413b2ae"
format_stamp: "Formatted at 2026-06-20 02:28:08 on dist-test-slave-r706"
server_key: "0bd15b5c73d78421f6cf90fd44b45a4a"
server_key_iv: "e15c7be78753d2e11d824aa1b575a042"
server_key_version: "encryptionkey@0"
I20260620 02:28:08.981698 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:28:08.986243 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:08.987831 501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:28:08.988075 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "75b22fa3e6e2462f8979b45ff413b2ae"
format_stamp: "Formatted at 2026-06-20 02:28:08 on dist-test-slave-r706"
server_key: "0bd15b5c73d78421f6cf90fd44b45a4a"
server_key_iv: "e15c7be78753d2e11d824aa1b575a042"
server_key_version: "encryptionkey@0"
I20260620 02:28:08.988336 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:09.033202 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:09.039247 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:09.039701 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:09.041401 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:09.043373 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:09.043525 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:09.043720 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:09.043828 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:09.115249 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46061
I20260620 02:28:09.115370 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46061 every 8 connection(s)
I20260620 02:28:09.116866 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:09.123032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 501
I20260620 02:28:09.123484 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:09.125203 388 external_mini_cluster.cc:1428] Setting key 21fb717659fdae0bdce5bad76e9e7060
Jun 20 02:28:09 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:28:09.151501 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:09.120826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58311 (local address 127.0.97.62:39791)
0620 02:28:09.121689 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:09.121694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:09.123104 (+ 1410us) server_negotiation.cc:407] Connection header received
0620 02:28:09.126875 (+ 3771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:09.126883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:09.127038 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:09.127225 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:09.129574 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:09.130634 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:09.135607 (+ 4973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:09.136145 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:09.136263 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:09.142994 (+ 6731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:09.143004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:09.143013 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:09.143077 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:09.147603 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:09.148503 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:09.148508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:09.148510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:09.148674 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:09.149276 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:09.149280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:09.149283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:09.149643 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:09.149839 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:09.150641 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 02:28:09.150909 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":499,"threads_started":1}
I20260620 02:28:09.152698 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:09.121504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39791 (local address 127.0.97.1:58311)
0620 02:28:09.122749 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:09.122845 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:09.126452 (+ 3607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:09.127595 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:09.127604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:09.128276 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:09.129362 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:09.129401 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:09.130882 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:09.130888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:09.135217 (+ 4329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:09.135283 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:09.136517 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:09.136533 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:09.137140 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:09.138209 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:09.138257 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:09.142513 (+ 4256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:09.147782 (+ 5269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:09.147794 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:09.147820 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:09.148339 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:09.148802 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:09.148807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:09.148811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:09.149141 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:09.149814 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:09.149824 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:09.150199 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 02:28:09.150531 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 02:28:09.151176 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":581,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:28:09.154829 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39791
I20260620 02:28:09.155330 629 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:09.156615 629 heartbeater.cc:511] Master 127.0.97.62:39791 requested a full tablet report, sending...
I20260620 02:28:09.159806 440 ts_manager.cc:194] Registered new tserver with Master: 75b22fa3e6e2462f8979b45ff413b2ae (127.0.97.1:46061)
I20260620 02:28:09.161942 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:09.181994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39791
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:09.706754 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:09.707130 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:09.707289 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:09.716616 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:09.716794 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:09.716965 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:09.729386 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:09.730880 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:09.732453 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:09.750872 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:09.751070 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:09.751602 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:09.752462 635 server_base.cc:1034] running on GCE node
I20260620 02:28:09.753310 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:09.754395 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:09.755636 635 hybrid_clock.cc:648] HybridClock initialized: now 1781922489755511 us; error 77 us; skew 500 ppm
I20260620 02:28:09.756138 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:09 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:09.760454 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:09.762590 635 webserver.cc:460] Webserver started at http://127.0.97.2:34755/ using document root <none> and password file <none>
I20260620 02:28:09.763429 635 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:09.763620 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:09.764006 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:09.767606 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "92c55819121c4fb48abda7ea40d575fc"
format_stamp: "Formatted at 2026-06-20 02:28:09 on dist-test-slave-r706"
server_key: "1ffb171050ad6dc53d84ddfb64d3c6ca"
server_key_iv: "9de59c2a7ec251aaaf421e8bcf196528"
server_key_version: "encryptionkey@0"
I20260620 02:28:09.768597 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "92c55819121c4fb48abda7ea40d575fc"
format_stamp: "Formatted at 2026-06-20 02:28:09 on dist-test-slave-r706"
server_key: "1ffb171050ad6dc53d84ddfb64d3c6ca"
server_key_iv: "9de59c2a7ec251aaaf421e8bcf196528"
server_key_version: "encryptionkey@0"
I20260620 02:28:09.774968 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:28:09.779639 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:09.781304 635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:28:09.781520 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "92c55819121c4fb48abda7ea40d575fc"
format_stamp: "Formatted at 2026-06-20 02:28:09 on dist-test-slave-r706"
server_key: "1ffb171050ad6dc53d84ddfb64d3c6ca"
server_key_iv: "9de59c2a7ec251aaaf421e8bcf196528"
server_key_version: "encryptionkey@0"
I20260620 02:28:09.781787 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:09.813684 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:09.819097 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:09.819561 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:09.820875 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:09.822919 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:09.823078 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:09.823288 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:09.823405 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:10.028966 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38349
I20260620 02:28:10.029325 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38349 every 8 connection(s)
I20260620 02:28:10.037181 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:10.041978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 635
I20260620 02:28:10.043340 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:10.047788 388 external_mini_cluster.cc:1428] Setting key 35d13d3a7a8747ef17aef7d14ef9ece0
Jun 20 02:28:10 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:28:10.165980 629 heartbeater.cc:503] Master 127.0.97.62:39791 was elected leader, sending a full tablet report...
I20260620 02:28:10.186815 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:10.054862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57113 (local address 127.0.97.62:39791)
0620 02:28:10.059388 (+ 4526us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:10.059467 (+ 79us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:10.069260 (+ 9793us) server_negotiation.cc:407] Connection header received
0620 02:28:10.105409 (+ 36149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:10.105415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:10.105645 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:10.106156 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:10.115054 (+ 8898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:10.118088 (+ 3034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:10.124608 (+ 6520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:10.126455 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:10.126935 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:10.146383 (+ 19448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:10.146395 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:10.146401 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:10.146470 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:10.168223 (+ 21753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:10.172631 (+ 4408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:10.172684 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:10.172690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:10.173258 (+ 568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:10.175327 (+ 2069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:10.175374 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:10.175401 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:10.176822 (+ 1421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:10.177908 (+ 1086us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:10.182173 (+ 4265us) server_negotiation.cc:299] Negotiation successful
0620 02:28:10.183663 (+ 1490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3451,"thread_start_us":2735,"threads_started":1}
I20260620 02:28:10.191053 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:10.058940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39791 (local address 127.0.97.2:57113)
0620 02:28:10.067334 (+ 8394us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:10.067982 (+ 648us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:10.102747 (+ 34765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:10.107614 (+ 4867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:10.107640 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:10.111317 (+ 3677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:10.113985 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:10.114065 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:10.118900 (+ 4835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:10.118918 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:10.123858 (+ 4940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:10.123926 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:10.127458 (+ 3532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:10.127507 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:10.128495 (+ 988us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:10.130644 (+ 2149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:10.130755 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:10.144312 (+ 13557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:10.168976 (+ 24664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:10.169161 (+ 185us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:10.169233 (+ 72us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:10.172044 (+ 2811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:10.173609 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:10.173614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:10.173625 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:10.174509 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:10.177623 (+ 3114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:10.177702 (+ 79us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:10.179495 (+ 1793us) client_negotiation.cc:769] Sending connection context
0620 02:28:10.181284 (+ 1789us) client_negotiation.cc:240] Negotiation successful
0620 02:28:10.184414 (+ 3130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5290,"thread_start_us":3132,"threads_started":1}
I20260620 02:28:10.199994 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39791
I20260620 02:28:10.200711 763 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:10.204126 763 heartbeater.cc:511] Master 127.0.97.62:39791 requested a full tablet report, sending...
I20260620 02:28:10.213141 440 ts_manager.cc:194] Registered new tserver with Master: 92c55819121c4fb48abda7ea40d575fc (127.0.97.2:38349)
I20260620 02:28:10.216941 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260620 02:28:10.268658 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.208s user 0.007s sys 0.073s
I20260620 02:28:10.274188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39791
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:10.868499 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:10.868862 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:10.869058 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:10.878163 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:10.878331 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:10.878482 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:10.890733 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:10.892241 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:10.893919 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:10.912350 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:10.912251 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:10.913404 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:10.915014 769 server_base.cc:1034] running on GCE node
I20260620 02:28:10.916003 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:10.917917 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:10.919375 769 hybrid_clock.cc:648] HybridClock initialized: now 1781922490919170 us; error 199 us; skew 500 ppm
I20260620 02:28:10.920081 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:10 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:10.927983 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:10.930454 769 webserver.cc:460] Webserver started at http://127.0.97.3:39141/ using document root <none> and password file <none>
I20260620 02:28:10.931404 769 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:10.931690 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:10.932328 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:10.938597 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "de28b80704d1402598bcca11e8152c3f"
format_stamp: "Formatted at 2026-06-20 02:28:10 on dist-test-slave-r706"
server_key: "5a6599ba37541b77832c429fc277fee8"
server_key_iv: "9400f9e1b63bbec75cee3ea9b15c2a07"
server_key_version: "encryptionkey@0"
I20260620 02:28:10.940090 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "de28b80704d1402598bcca11e8152c3f"
format_stamp: "Formatted at 2026-06-20 02:28:10 on dist-test-slave-r706"
server_key: "5a6599ba37541b77832c429fc277fee8"
server_key_iv: "9400f9e1b63bbec75cee3ea9b15c2a07"
server_key_version: "encryptionkey@0"
I20260620 02:28:10.950778 769 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.002s
I20260620 02:28:10.957741 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:10.959522 769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:28:10.959766 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "de28b80704d1402598bcca11e8152c3f"
format_stamp: "Formatted at 2026-06-20 02:28:10 on dist-test-slave-r706"
server_key: "5a6599ba37541b77832c429fc277fee8"
server_key_iv: "9400f9e1b63bbec75cee3ea9b15c2a07"
server_key_version: "encryptionkey@0"
I20260620 02:28:10.960079 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:10.991060 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:10.996680 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:10.997141 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:10.998387 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:11.000043 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:11.000164 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:11.000362 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:11.000483 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:11.067050 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35335
I20260620 02:28:11.067302 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35335 every 8 connection(s)
I20260620 02:28:11.068651 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:11.077584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 769
I20260620 02:28:11.077821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:11.078388 388 external_mini_cluster.cc:1428] Setting key 704fb3901d7e315da90668b5e85dd4c2
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:11.097824 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.072316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48951 (local address 127.0.97.62:39791)
0620 02:28:11.073063 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:11.073070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:11.074411 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 02:28:11.076915 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:11.076922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:11.077103 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:11.077351 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:11.079637 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.081188 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.083992 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.084532 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.084661 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.089436 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:11.089442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:11.089446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:11.089504 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:11.093453 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.094737 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.094741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.094744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.094992 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.095676 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.095681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.095684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.096024 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:11.096192 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:11.097185 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 02:28:11.097392 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":430,"threads_started":1}
I20260620 02:28:11.098997 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.073113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39791 (local address 127.0.97.3:48951)
0620 02:28:11.074179 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:11.074227 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:11.076559 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:11.077656 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:11.077674 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:11.078536 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:11.079363 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:11.079393 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.081418 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.081422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:11.083852 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:11.083866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.084825 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.084832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:11.085207 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.086031 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:11.086056 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:11.089050 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:11.093766 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:11.093775 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:11.093793 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:11.094585 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:11.095151 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:11.095155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:11.095158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:11.095518 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:11.096234 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:11.096243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:11.096726 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 02:28:11.097106 (+ 380us) client_negotiation.cc:240] Negotiation successful
0620 02:28:11.097589 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":526,"threads_started":1}
I20260620 02:28:11.100693 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39791
I20260620 02:28:11.101087 897 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:11.102108 897 heartbeater.cc:511] Master 127.0.97.62:39791 requested a full tablet report, sending...
I20260620 02:28:11.103948 440 ts_manager.cc:194] Registered new tserver with Master: de28b80704d1402598bcca11e8152c3f (127.0.97.3:35335)
I20260620 02:28:11.105348 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48951
I20260620 02:28:11.109515 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:11.174748 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.150783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.0.97.1:46061)
0620 02:28:11.152361 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:11.152371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:11.152418 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:28:11.152629 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:11.152639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:11.152954 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:11.153253 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:11.156064 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.157617 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.159308 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.160059 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.160224 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.166150 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:11.166165 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:11.166181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:11.166263 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:11.170916 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.171757 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.171769 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.171773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.172002 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.172464 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.172468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.172470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.172965 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:11.173463 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:11.173780 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:28:11.174202 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":812,"threads_started":1}
W20260620 02:28:11.176467 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49568
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:11.228128 763 heartbeater.cc:503] Master 127.0.97.62:39791 was elected leader, sending a full tablet report...
I20260620 02:28:11.228837 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.210472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.0.97.1:46061)
0620 02:28:11.210753 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:11.210764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:11.211036 (+ 272us) server_negotiation.cc:407] Connection header received
0620 02:28:11.211318 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:11.211323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:11.211430 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:11.211567 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:11.213461 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.214277 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.215865 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.216493 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.216667 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.221865 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:11.221871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:11.221874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:11.221915 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:11.225978 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.226640 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.226645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.226647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.226839 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.227270 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.227274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.227276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.227546 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:11.227780 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:11.228197 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 02:28:11.228460 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 02:28:11.230806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 501
I20260620 02:28:11.247800 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.231839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.0.97.1:46061)
0620 02:28:11.232005 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:11.232009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:11.232604 (+ 595us) server_negotiation.cc:407] Connection header received
0620 02:28:11.232718 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:11.232722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:11.232780 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:11.232860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:11.234621 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.235411 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.236964 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.237610 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.237717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.240824 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:11.240829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:11.240833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:11.240868 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:11.244701 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.245652 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.245656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.245659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.245857 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.246258 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.246261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.246263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.246569 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:11.246806 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:11.247196 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 02:28:11.247379 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:28:11.248914 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:11.374429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 501
I20260620 02:28:11.391407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 635
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:11.409322 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.392489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.0.97.2:38349)
0620 02:28:11.393305 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:11.393312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:11.393340 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:28:11.393413 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:11.393420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:11.393652 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:11.393805 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:11.395082 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.396193 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.397677 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.398284 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.398391 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.402705 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:11.402717 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:11.402732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:11.402790 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:11.406487 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.407167 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.407174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.407176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.407323 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.407780 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.407783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.407785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.408169 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:11.408467 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:11.408731 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 02:28:11.408951 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":354,"threads_started":1}
I20260620 02:28:11.410748 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:11.533857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 635
I20260620 02:28:11.557220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 769
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:11.574967 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:11.558371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.0.97.3:35335)
0620 02:28:11.558957 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:11.558964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:11.558994 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:28:11.559054 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:11.559060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:11.559271 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:11.559455 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:11.560799 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.561989 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.563342 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:11.563921 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:11.564069 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:11.568585 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:11.568596 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:11.568617 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:11.568656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:11.571906 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.572670 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.572677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.572680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.572808 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:11.573428 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:11.573432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:11.573434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:11.573801 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:11.574032 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:11.574466 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 02:28:11.574682 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":304,"threads_started":1}
I20260620 02:28:11.576298 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:11.661177 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 769
I20260620 02:28:11.677778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 408
I20260620 02:28:11.679145 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:11.829037 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 408
2026-06-20T02:28:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6279 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[919](info): set up 2 sockets
Jun 20 02:28:11 dist-test-slave-r706 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260620 02:28:12.928875 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 02:28:13.072467 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.143s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 02:28:13.207360 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.135s user 0.001s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:13 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:13.393795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42721
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:13.728641 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:13.728945 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:13.729131 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:13.736712 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:13.736824 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:13.736912 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:13.737037 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:13.737100 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:13.747364 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:13.748801 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:13.750322 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:13.762431 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:13.762476 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:13.762511 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:13.763901 932 server_base.cc:1034] running on GCE node
I20260620 02:28:13.764648 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:13.766160 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:13.767392 932 hybrid_clock.cc:648] HybridClock initialized: now 1781922493767309 us; error 62 us; skew 500 ppm
I20260620 02:28:13.767804 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:13 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:13.771159 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:13.772861 932 webserver.cc:460] Webserver started at http://127.0.97.62:38415/ using document root <none> and password file <none>
I20260620 02:28:13.773655 932 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:13.773813 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:13.774148 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:13.777245 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "c2a944f5a5624212b9a47bafad8ebeb7"
format_stamp: "Formatted at 2026-06-20 02:28:13 on dist-test-slave-r706"
server_key: "9f064a56c1c9a67ae6c3e7b9dffc4894"
server_key_iv: "c7c98e1ce927dbcfc1644133ca6d7294"
server_key_version: "encryptionkey@0"
I20260620 02:28:13.778004 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2a944f5a5624212b9a47bafad8ebeb7"
format_stamp: "Formatted at 2026-06-20 02:28:13 on dist-test-slave-r706"
server_key: "9f064a56c1c9a67ae6c3e7b9dffc4894"
server_key_iv: "c7c98e1ce927dbcfc1644133ca6d7294"
server_key_version: "encryptionkey@0"
I20260620 02:28:13.783357 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:13.786921 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:13.788327 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:13.788551 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "c2a944f5a5624212b9a47bafad8ebeb7"
format_stamp: "Formatted at 2026-06-20 02:28:13 on dist-test-slave-r706"
server_key: "9f064a56c1c9a67ae6c3e7b9dffc4894"
server_key_iv: "c7c98e1ce927dbcfc1644133ca6d7294"
server_key_version: "encryptionkey@0"
I20260620 02:28:13.788776 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:13.816597 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:13.821305 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:13.821597 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:13.842306 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42721
I20260620 02:28:13.842355 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42721 every 8 connection(s)
I20260620 02:28:13.843791 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:13.847699 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:13.852515 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 932
I20260620 02:28:13.852734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:13.853639 388 external_mini_cluster.cc:1428] Setting key b52c607cebe38c50cce9cd93f5d662be
I20260620 02:28:13.860126 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7: Bootstrap starting.
I20260620 02:28:13.863930 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:13.865319 999 log.cc:824] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:13.868961 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7: No bootstrap required, opened a new log
Jun 20 02:28:13 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:13.877140 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42721 } }
I20260620 02:28:13.877579 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:13.877768 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a944f5a5624212b9a47bafad8ebeb7, State: Initialized, Role: FOLLOWER
I20260620 02:28:13.878355 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42721 } }
I20260620 02:28:13.878639 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:13.855882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.97.62:42721)
0620 02:28:13.856861 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:13.856873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:13.856924 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:28:13.858847 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:13.858870 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:13.859276 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:13.859633 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:13.861600 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:13.863121 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:13.864977 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:13.865790 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:13.866047 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:13.870755 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:13.870763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:13.870776 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:13.870815 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:13.874153 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:13.874859 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:13.874866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:13.874871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:13.875051 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:13.875435 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:13.875443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:13.875450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:13.875993 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:13.876322 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:13.876803 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 02:28:13.877268 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":530,"threads_started":1}
I20260620 02:28:13.878654 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:13.878891 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:13.879078 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:13.881584 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42721 } }
I20260620 02:28:13.882129 999 leader_election.cc:302] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2a944f5a5624212b9a47bafad8ebeb7; no voters:
I20260620 02:28:13.882747 999 leader_election.cc:288] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:13.882895 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:13.883610 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [term 1 LEADER]: Becoming Leader. State: Replica: c2a944f5a5624212b9a47bafad8ebeb7, State: Running, Role: LEADER
I20260620 02:28:13.884150 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42721 } }
I20260620 02:28:13.884876 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:13.887255 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2a944f5a5624212b9a47bafad8ebeb7. Latest consensus state: current_term: 1 leader_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42721 } } }
I20260620 02:28:13.887245 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a944f5a5624212b9a47bafad8ebeb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42721 } } }
I20260620 02:28:13.887722 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:13.888346 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:13.889458 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:13.895584 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:13.904739 1009 catalog_manager.cc:1347] Generated new cluster ID: 08a720e03a8447d5912f4d5d4b568be4
I20260620 02:28:13.904874 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:13.919603 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:13.921139 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:13.929586 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2a944f5a5624212b9a47bafad8ebeb7: Generated new TSK 0
I20260620 02:28:13.930537 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:13.990264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42721
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:14.344641 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:14.344923 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:14.345085 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:14.352398 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:14.352521 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:14.352639 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:14.362596 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42721
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:14.363843 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:14.365150 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:14.379357 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:14.379398 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:14.379323 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:14.380241 1025 server_base.cc:1034] running on GCE node
I20260620 02:28:14.380961 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:14.382035 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:14.383286 1025 hybrid_clock.cc:648] HybridClock initialized: now 1781922494383211 us; error 86 us; skew 500 ppm
I20260620 02:28:14.383685 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:14 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:14.387169 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:14.388880 1025 webserver.cc:460] Webserver started at http://127.0.97.1:42281/ using document root <none> and password file <none>
I20260620 02:28:14.389653 1025 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:14.389773 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:14.390092 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:14.393740 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "589ef2a8c481408f9d63139babdb3529"
format_stamp: "Formatted at 2026-06-20 02:28:14 on dist-test-slave-r706"
server_key: "068f170eea094873d7ef18504c8a12d4"
server_key_iv: "0de31d622b3c9f6477a96b8f700720de"
server_key_version: "encryptionkey@0"
I20260620 02:28:14.394631 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "589ef2a8c481408f9d63139babdb3529"
format_stamp: "Formatted at 2026-06-20 02:28:14 on dist-test-slave-r706"
server_key: "068f170eea094873d7ef18504c8a12d4"
server_key_iv: "0de31d622b3c9f6477a96b8f700720de"
server_key_version: "encryptionkey@0"
I20260620 02:28:14.400303 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:28:14.404062 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:14.405490 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:28:14.405670 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "589ef2a8c481408f9d63139babdb3529"
format_stamp: "Formatted at 2026-06-20 02:28:14 on dist-test-slave-r706"
server_key: "068f170eea094873d7ef18504c8a12d4"
server_key_iv: "0de31d622b3c9f6477a96b8f700720de"
server_key_version: "encryptionkey@0"
I20260620 02:28:14.405879 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:14.428647 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:14.433614 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:14.433874 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:14.434805 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:14.436091 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:14.436167 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:14.436250 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:14.436304 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:14.469345 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35039
I20260620 02:28:14.469408 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35039 every 8 connection(s)
I20260620 02:28:14.470678 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:14.472148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1025
I20260620 02:28:14.472349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:14.473181 388 external_mini_cluster.cc:1428] Setting key 2ca53d24c0236259fdc5327a66a038fe
Jun 20 02:28:14 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:28:14.493558 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:14.473748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45143 (local address 127.0.97.62:42721)
0620 02:28:14.474276 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:14.474282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:14.475432 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 02:28:14.477441 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:14.477447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:14.477539 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:14.477661 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:14.479250 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:14.479989 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:14.482127 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:14.482552 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:14.482668 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:14.487509 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:14.487514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:14.487518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:14.487550 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:14.490216 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:14.490953 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:14.490960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:14.490963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:14.491091 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:14.491536 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:14.491541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:14.491545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:14.491893 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:14.492054 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:14.492856 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 02:28:14.493112 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:28:14.494688 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:14.474381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42721 (local address 127.0.97.1:45143)
0620 02:28:14.475243 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:14.475302 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:14.477157 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:14.477928 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:14.477937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:14.478472 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:14.479062 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:14.479081 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:14.480174 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:14.480179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:14.481927 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:14.481946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:14.482803 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:14.482813 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:14.483168 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:14.483970 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:14.484006 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:14.487221 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:14.490367 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:14.490376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:14.490394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:14.490822 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:14.491204 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:14.491209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:14.491213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:14.491412 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:14.492044 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:14.492052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:14.492468 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 02:28:14.492735 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 02:28:14.493331 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":324,"threads_started":1}
I20260620 02:28:14.496599 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42721
I20260620 02:28:14.496989 1153 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:14.498077 1153 heartbeater.cc:511] Master 127.0.97.62:42721 requested a full tablet report, sending...
I20260620 02:28:14.500578 964 ts_manager.cc:194] Registered new tserver with Master: 589ef2a8c481408f9d63139babdb3529 (127.0.97.1:35039)
I20260620 02:28:14.502983 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:14.521059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42721
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:14.865581 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:14.865876 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:14.865975 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:14.873637 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:14.873759 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:14.873880 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:14.883909 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42721
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:14.885488 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:14.886945 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:14.900696 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:14.900705 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:14.901214 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:14.902064 1159 server_base.cc:1034] running on GCE node
I20260620 02:28:14.902631 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:14.903669 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:14.904912 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781922494904842 us; error 75 us; skew 500 ppm
I20260620 02:28:14.905346 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:14 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:14.908699 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:14.910432 1159 webserver.cc:460] Webserver started at http://127.0.97.2:42013/ using document root <none> and password file <none>
I20260620 02:28:14.911237 1159 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:14.911381 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:14.911720 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:14.915055 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "3116a967287b4c04a82158a55f2c64f3"
format_stamp: "Formatted at 2026-06-20 02:28:14 on dist-test-slave-r706"
server_key: "44df7a2cd36e87868bbe05e452951049"
server_key_iv: "f07dc5074abc48920f8346649359fdaf"
server_key_version: "encryptionkey@0"
I20260620 02:28:14.915957 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3116a967287b4c04a82158a55f2c64f3"
format_stamp: "Formatted at 2026-06-20 02:28:14 on dist-test-slave-r706"
server_key: "44df7a2cd36e87868bbe05e452951049"
server_key_iv: "f07dc5074abc48920f8346649359fdaf"
server_key_version: "encryptionkey@0"
I20260620 02:28:14.921334 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:28:14.924886 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:14.926246 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:14.926453 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "3116a967287b4c04a82158a55f2c64f3"
format_stamp: "Formatted at 2026-06-20 02:28:14 on dist-test-slave-r706"
server_key: "44df7a2cd36e87868bbe05e452951049"
server_key_iv: "f07dc5074abc48920f8346649359fdaf"
server_key_version: "encryptionkey@0"
I20260620 02:28:14.926662 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:14.939765 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:14.944463 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:14.944800 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:14.945883 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:14.947324 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:14.947434 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:14.947595 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:14.947671 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:14.989157 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45967
I20260620 02:28:14.989218 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45967 every 8 connection(s)
I20260620 02:28:14.990578 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:14.992815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1159
I20260620 02:28:14.992988 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:14.993535 388 external_mini_cluster.cc:1428] Setting key 6ef55006f944adaca1942fce78bf3a63
Jun 20 02:28:15 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:28:15.014726 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:14.994555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34505 (local address 127.0.97.62:42721)
0620 02:28:14.995008 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:14.995014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:14.995425 (+ 411us) server_negotiation.cc:407] Connection header received
0620 02:28:14.997264 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:14.997273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:14.997368 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:14.997496 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:14.999000 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.000231 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.002758 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.003213 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.003318 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.008269 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:15.008278 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:15.008285 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:15.008332 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:15.011239 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.012240 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.012244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.012246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.012375 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.012914 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.012918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.012920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.013256 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:15.013375 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.014185 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.014359 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":993,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:28:15.015926 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:14.994118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42721 (local address 127.0.97.2:34505)
0620 02:28:14.994992 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:14.995051 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:14.996917 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:14.997694 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:14.997702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:14.998238 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:14.998823 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:14.998841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.000389 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.000393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.002584 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.002603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.003455 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.003473 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.003820 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.004585 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:15.004620 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:15.007973 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:15.011384 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:15.011398 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:15.011419 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:15.012102 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:15.012529 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:15.012535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:15.012540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:15.012794 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:15.013403 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:15.013409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:15.013825 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.014074 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.014565 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":514,"threads_started":1}
I20260620 02:28:15.017563 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42721
I20260620 02:28:15.017954 1287 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:15.018903 1287 heartbeater.cc:511] Master 127.0.97.62:42721 requested a full tablet report, sending...
I20260620 02:28:15.020646 964 ts_manager.cc:194] Registered new tserver with Master: 3116a967287b4c04a82158a55f2c64f3 (127.0.97.2:45967)
I20260620 02:28:15.021556 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:15.040758 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42721
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:15.379192 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:15.379539 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:15.379702 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:15.387240 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:15.387404 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:15.387600 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:15.398175 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42721
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:15.399516 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:15.400866 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:15.414729 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:15.414659 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:15.414616 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:15.415791 1293 server_base.cc:1034] running on GCE node
I20260620 02:28:15.416500 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:15.417614 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:15.418895 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781922495418818 us; error 53 us; skew 500 ppm
I20260620 02:28:15.419246 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:15 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:15.422430 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:15.423915 1293 webserver.cc:460] Webserver started at http://127.0.97.3:36523/ using document root <none> and password file <none>
I20260620 02:28:15.424692 1293 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:15.424808 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:15.425139 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:15.428197 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "3cab722c967046d08757efe62d4933f5"
format_stamp: "Formatted at 2026-06-20 02:28:15 on dist-test-slave-r706"
server_key: "01f5183a342d1c863ad3e59358026e0b"
server_key_iv: "50a136c183d3092091479694ff644990"
server_key_version: "encryptionkey@0"
I20260620 02:28:15.428987 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3cab722c967046d08757efe62d4933f5"
format_stamp: "Formatted at 2026-06-20 02:28:15 on dist-test-slave-r706"
server_key: "01f5183a342d1c863ad3e59358026e0b"
server_key_iv: "50a136c183d3092091479694ff644990"
server_key_version: "encryptionkey@0"
I20260620 02:28:15.434981 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:15.439006 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:15.440485 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:15.440634 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "3cab722c967046d08757efe62d4933f5"
format_stamp: "Formatted at 2026-06-20 02:28:15 on dist-test-slave-r706"
server_key: "01f5183a342d1c863ad3e59358026e0b"
server_key_iv: "50a136c183d3092091479694ff644990"
server_key_version: "encryptionkey@0"
I20260620 02:28:15.440819 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:15.459904 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:15.464547 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:15.464828 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:15.465924 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:15.467270 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:15.467353 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:15.467442 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:15.467487 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:15.506289 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41171
I20260620 02:28:15.506429 1153 heartbeater.cc:503] Master 127.0.97.62:42721 was elected leader, sending a full tablet report...
I20260620 02:28:15.506326 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41171 every 8 connection(s)
I20260620 02:28:15.507682 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:15.509691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1293
I20260620 02:28:15.509886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:15.510362 388 external_mini_cluster.cc:1428] Setting key 2bdf32101e0736ac10f9cfb972284421
Jun 20 02:28:15 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:15.532709 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.510573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33281 (local address 127.0.97.62:42721)
0620 02:28:15.510802 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.510809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.512463 (+ 1654us) server_negotiation.cc:407] Connection header received
0620 02:28:15.515274 (+ 2811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.515278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.515383 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.515485 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:15.517413 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.518202 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.519962 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.520391 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.520488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.524563 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:15.524569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:15.524573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:15.524607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:15.528575 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.529653 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.529661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.529666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.529908 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.530489 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.530495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.530499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.530866 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:15.531047 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.531894 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.532201 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:28:15.533957 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.511260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42721 (local address 127.0.97.3:33281)
0620 02:28:15.512278 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:15.512329 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:15.514970 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:15.515746 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:15.515755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:15.516407 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:15.517239 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.517259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.518336 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.518339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.519822 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.519833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.520643 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.520648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.520907 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.521659 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:15.521696 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:15.524311 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:15.528829 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:15.528841 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:15.528860 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:15.529492 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:15.530059 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:15.530065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:15.530068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:15.530352 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:15.531034 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:15.531043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:15.531450 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.531778 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.532456 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":481,"threads_started":1}
I20260620 02:28:15.535524 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42721
I20260620 02:28:15.535883 1421 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:15.536815 1421 heartbeater.cc:511] Master 127.0.97.62:42721 requested a full tablet report, sending...
I20260620 02:28:15.538756 964 ts_manager.cc:194] Registered new tserver with Master: 3cab722c967046d08757efe62d4933f5 (127.0.97.3:41171)
I20260620 02:28:15.539984 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:28:15.539943 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33281
I20260620 02:28:15.564766 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.548367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.0.97.62:42721)
0620 02:28:15.548647 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.548655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.549342 (+ 687us) server_negotiation.cc:407] Connection header received
0620 02:28:15.549496 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.549500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.549567 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.549664 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:15.551737 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.552708 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.554385 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.554881 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.554996 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.558149 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:15.558154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:15.558158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:15.558196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:15.561444 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.562197 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.562201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.562203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.562397 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.562834 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.562838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.562841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.563310 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:15.563555 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.563988 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.564237 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:28:15.572480 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:15.581748 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:15.616154 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.598372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.0.97.3:41171)
0620 02:28:15.603181 (+ 4809us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.603194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.604303 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 02:28:15.604409 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.604417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.604800 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.605095 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:15.606702 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.609207 (+ 2505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.612958 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.614445 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.614602 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.614970 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.615365 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.615597 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4550,"thread_start_us":4288,"threads_started":1}
I20260620 02:28:15.616289 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.597508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41171 (local address 127.0.0.1:55294)
0620 02:28:15.603994 (+ 6486us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:15.604011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:15.604145 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:15.605521 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:15.605528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:15.605568 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:15.606321 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.606338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.609411 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.609419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.612701 (+ 3282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.612718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.614711 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.614718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.615063 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.615102 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.615287 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.615544 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6349,"mutex_wait_us":31,"thread_start_us":6137,"threads_started":1}
I20260620 02:28:15.616845 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.596752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45967 (local address 127.0.0.1:38774)
0620 02:28:15.598263 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:15.598314 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:15.598559 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:15.606028 (+ 7469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:15.606037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:15.606082 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:15.607035 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.607052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.609970 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.609976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.612942 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.612965 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.615497 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.615504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.615726 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.615798 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.615878 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.616039 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":89,"thread_start_us":924,"threads_started":1}
I20260620 02:28:15.618222 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.597412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.97.2:45967)
0620 02:28:15.605240 (+ 7828us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.605250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.605300 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:28:15.605419 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.605427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.605749 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.606011 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:15.607506 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.609583 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.613208 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.615177 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.617248 (+ 2071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.617435 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.617507 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.617714 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7614,"thread_start_us":7448,"threads_started":1}
I20260620 02:28:15.619422 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.596956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35039 (local address 127.0.0.1:48920)
0620 02:28:15.601867 (+ 4911us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:15.601892 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:15.602042 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:15.607990 (+ 5948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:15.607995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:15.608020 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:15.608645 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.608657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.611663 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.611668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.615073 (+ 3405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.615088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.618281 (+ 3193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.618288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.618511 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.618541 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.618658 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.618786 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4709,"mutex_wait_us":27,"thread_start_us":4470,"threads_started":1}
I20260620 02:28:15.619422 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.597431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.0.97.1:35039)
0620 02:28:15.607317 (+ 9886us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.607325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.607361 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:28:15.607485 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.607493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.607763 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.608843 (+ 1080us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:15.609248 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.611487 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.615274 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.617996 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.618200 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.618667 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.618733 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.618955 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9619,"thread_start_us":9308,"threads_started":1}
I20260620 02:28:15.621129 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 91c8a7ffe02d4146b91ae65837ae38a2 (DEFAULT_TABLE table=test-table [id=c877633e355e46a9beb21c121e8ecee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:15.623359 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c8a7ffe02d4146b91ae65837ae38a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:15.623953 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 91c8a7ffe02d4146b91ae65837ae38a2 (DEFAULT_TABLE table=test-table [id=c877633e355e46a9beb21c121e8ecee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:15.624722 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 91c8a7ffe02d4146b91ae65837ae38a2 (DEFAULT_TABLE table=test-table [id=c877633e355e46a9beb21c121e8ecee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:15.625705 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c8a7ffe02d4146b91ae65837ae38a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:15.626058 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c8a7ffe02d4146b91ae65837ae38a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:15.641975 1439 tablet_bootstrap.cc:492] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: Bootstrap starting.
I20260620 02:28:15.645798 1440 tablet_bootstrap.cc:492] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: Bootstrap starting.
I20260620 02:28:15.649631 1439 tablet_bootstrap.cc:654] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:15.650950 1441 tablet_bootstrap.cc:492] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: Bootstrap starting.
I20260620 02:28:15.651541 1440 tablet_bootstrap.cc:654] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:15.651830 1439 log.cc:824] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:15.653802 1440 log.cc:824] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:15.655831 1441 tablet_bootstrap.cc:654] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:15.657132 1439 tablet_bootstrap.cc:492] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: No bootstrap required, opened a new log
I20260620 02:28:15.657531 1439 ts_tablet_manager.cc:1397] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260620 02:28:15.658025 1441 log.cc:824] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:15.660303 1440 tablet_bootstrap.cc:492] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: No bootstrap required, opened a new log
I20260620 02:28:15.660738 1440 ts_tablet_manager.cc:1397] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 02:28:15.664508 1441 tablet_bootstrap.cc:492] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: No bootstrap required, opened a new log
I20260620 02:28:15.664921 1441 ts_tablet_manager.cc:1397] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260620 02:28:15.668893 1439 raft_consensus.cc:348] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.669564 1439 raft_consensus.cc:374] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:15.669754 1439 raft_consensus.cc:729] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3116a967287b4c04a82158a55f2c64f3, State: Initialized, Role: FOLLOWER
I20260620 02:28:15.670538 1439 consensus_queue.cc:260] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.674358 1287 heartbeater.cc:503] Master 127.0.97.62:42721 was elected leader, sending a full tablet report...
I20260620 02:28:15.675316 1439 ts_tablet_manager.cc:1428] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260620 02:28:15.675621 1440 raft_consensus.cc:348] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.676262 1440 raft_consensus.cc:374] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:15.676403 1440 raft_consensus.cc:729] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cab722c967046d08757efe62d4933f5, State: Initialized, Role: FOLLOWER
I20260620 02:28:15.677284 1440 consensus_queue.cc:260] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.679208 1421 heartbeater.cc:503] Master 127.0.97.62:42721 was elected leader, sending a full tablet report...
I20260620 02:28:15.679916 1440 ts_tablet_manager.cc:1428] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20260620 02:28:15.680085 1441 raft_consensus.cc:348] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.680609 1441 raft_consensus.cc:374] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:15.680760 1441 raft_consensus.cc:729] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589ef2a8c481408f9d63139babdb3529, State: Initialized, Role: FOLLOWER
I20260620 02:28:15.681541 1441 consensus_queue.cc:260] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.683674 1441 ts_tablet_manager.cc:1428] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
W20260620 02:28:15.723340 1154 tablet.cc:2369] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:15.742959 1288 tablet.cc:2369] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:15.759654 1422 tablet.cc:2369] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:15.853964 1447 raft_consensus.cc:482] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:15.854383 1447 raft_consensus.cc:504] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.856590 1447 leader_election.cc:288] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3116a967287b4c04a82158a55f2c64f3 (127.0.97.2:45967), 3cab722c967046d08757efe62d4933f5 (127.0.97.3:41171)
I20260620 02:28:15.867035 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.857052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58993 (local address 127.0.97.3:41171)
0620 02:28:15.857361 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.857372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.859208 (+ 1836us) server_negotiation.cc:407] Connection header received
0620 02:28:15.859509 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.859516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.859620 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.859733 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:15.860804 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.861848 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.864497 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.865688 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.865798 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.865895 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.866504 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.866686 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:28:15.867112 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.856842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41171 (local address 127.0.97.1:58993)
0620 02:28:15.859039 (+ 2197us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:15.859070 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:15.859331 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:15.859807 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:15.859814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:15.859843 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:15.860595 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.860607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.861990 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.861995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.864289 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.864312 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.866032 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.866040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.866292 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.866331 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.866443 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.866550 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"mutex_wait_us":101,"thread_start_us":2131,"threads_started":1}
I20260620 02:28:15.867686 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.857874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36097 (local address 127.0.97.2:45967)
0620 02:28:15.858171 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.858179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.859699 (+ 1520us) server_negotiation.cc:407] Connection header received
0620 02:28:15.859880 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.859885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.860022 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.860169 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:15.861156 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.862193 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.864818 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.866073 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.866262 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.866355 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.866917 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.867306 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:28:15.867901 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.857297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45967 (local address 127.0.97.1:36097)
0620 02:28:15.859548 (+ 2251us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:15.859567 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:15.859732 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:15.860176 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:15.860179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:15.860203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:15.860925 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.860952 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.862377 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.862381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.864607 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:15.864621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.866472 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.866479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:15.866709 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.866742 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:28:15.866871 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 02:28:15.867033 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2110,"mutex_wait_us":4,"thread_start_us":911,"threads_started":1}
I20260620 02:28:15.868670 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c8a7ffe02d4146b91ae65837ae38a2" candidate_uuid: "589ef2a8c481408f9d63139babdb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cab722c967046d08757efe62d4933f5" is_pre_election: true
I20260620 02:28:15.869174 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c8a7ffe02d4146b91ae65837ae38a2" candidate_uuid: "589ef2a8c481408f9d63139babdb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3116a967287b4c04a82158a55f2c64f3" is_pre_election: true
I20260620 02:28:15.869369 1376 raft_consensus.cc:2454] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589ef2a8c481408f9d63139babdb3529 in term 0.
I20260620 02:28:15.869807 1242 raft_consensus.cc:2454] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589ef2a8c481408f9d63139babdb3529 in term 0.
I20260620 02:28:15.870410 1044 leader_election.cc:302] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cab722c967046d08757efe62d4933f5, 589ef2a8c481408f9d63139babdb3529; no voters:
I20260620 02:28:15.871014 1447 raft_consensus.cc:2781] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:15.871214 1447 raft_consensus.cc:482] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:15.871400 1447 raft_consensus.cc:3037] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:15.874383 1447 raft_consensus.cc:504] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.875217 1447 leader_election.cc:288] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [CANDIDATE]: Term 1 election: Requested vote from peers 3116a967287b4c04a82158a55f2c64f3 (127.0.97.2:45967), 3cab722c967046d08757efe62d4933f5 (127.0.97.3:41171)
I20260620 02:28:15.875777 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c8a7ffe02d4146b91ae65837ae38a2" candidate_uuid: "589ef2a8c481408f9d63139babdb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3116a967287b4c04a82158a55f2c64f3"
I20260620 02:28:15.875888 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c8a7ffe02d4146b91ae65837ae38a2" candidate_uuid: "589ef2a8c481408f9d63139babdb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cab722c967046d08757efe62d4933f5"
I20260620 02:28:15.876030 1242 raft_consensus.cc:3037] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:15.876178 1376 raft_consensus.cc:3037] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:15.879026 1242 raft_consensus.cc:2454] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3116a967287b4c04a82158a55f2c64f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589ef2a8c481408f9d63139babdb3529 in term 1.
I20260620 02:28:15.879093 1376 raft_consensus.cc:2454] T 91c8a7ffe02d4146b91ae65837ae38a2 P 3cab722c967046d08757efe62d4933f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589ef2a8c481408f9d63139babdb3529 in term 1.
I20260620 02:28:15.879539 1041 leader_election.cc:302] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3116a967287b4c04a82158a55f2c64f3, 589ef2a8c481408f9d63139babdb3529; no voters:
I20260620 02:28:15.879983 1447 raft_consensus.cc:2781] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:15.880703 1447 raft_consensus.cc:686] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [term 1 LEADER]: Becoming Leader. State: Replica: 589ef2a8c481408f9d63139babdb3529, State: Running, Role: LEADER
I20260620 02:28:15.881368 1447 consensus_queue.cc:237] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } }
I20260620 02:28:15.886768 964 catalog_manager.cc:5495] T 91c8a7ffe02d4146b91ae65837ae38a2 P 589ef2a8c481408f9d63139babdb3529 reported cstate change: term changed from 0 to 1, leader changed from <none> to 589ef2a8c481408f9d63139babdb3529 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "589ef2a8c481408f9d63139babdb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3116a967287b4c04a82158a55f2c64f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589ef2a8c481408f9d63139babdb3529" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cab722c967046d08757efe62d4933f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41171 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:28:15 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:15.931169 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.907419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.0.97.1:35039)
0620 02:28:15.907727 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.907740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.908643 (+ 903us) server_negotiation.cc:407] Connection header received
0620 02:28:15.908800 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.908804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.908870 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.908958 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:15.911340 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.912305 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.914067 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.914600 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.914721 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.923792 (+ 9071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:15.923811 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:15.923827 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:15.923885 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:15.927589 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.928459 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.928466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.928468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.928694 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.929300 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.929303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.929305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.929802 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:15.930071 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.930448 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.930843 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 02:28:15 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:28:15 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:15.986883 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:15.969320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.0.97.1:35039)
0620 02:28:15.969578 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:15.969592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:15.970126 (+ 534us) server_negotiation.cc:407] Connection header received
0620 02:28:15.970334 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:15.970340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:15.970423 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:15.970535 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:15.972340 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.973227 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.974865 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:15.975409 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:15.975556 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:15.980437 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:15.980448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:15.980451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:15.980486 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:15.983691 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.984387 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.984393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.984397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.984600 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:15.985074 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:15.985080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:15.985084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:15.985479 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:15.985881 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:15.986135 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:28:15.986425 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:16.043776 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:16.025612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.0.97.1:35039)
0620 02:28:16.025880 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:16.025889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:16.026352 (+ 463us) server_negotiation.cc:407] Connection header received
0620 02:28:16.026513 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:16.026518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:16.026600 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:16.026692 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:16.028698 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.029552 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.031419 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.031878 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.031993 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:16.037290 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:16.037296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:16.037299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:16.037340 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:16.040407 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.041433 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.041437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.041439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.041688 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.042153 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.042156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.042158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.042497 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:16.042793 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:16.043135 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:28:16.043344 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:28:16.046108 1068 tablet_service.cc:2908] Rejecting scan request for tablet 91c8a7ffe02d4146b91ae65837ae38a2: Uninitialized: clean time has not yet been initialized
I20260620 02:28:16.050103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1025
I20260620 02:28:16.065128 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:16.051129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.97.1:35039)
0620 02:28:16.051363 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:16.051369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:16.051658 (+ 289us) server_negotiation.cc:407] Connection header received
0620 02:28:16.051858 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:16.051881 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:16.051955 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:16.052066 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:16.053333 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.054094 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.055472 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.055937 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.056080 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:16.058865 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:16.058871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:16.058874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:16.058907 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:16.061957 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.062727 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.062734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.062738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.062910 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.063314 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.063320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.063324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.063632 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:16.063985 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:16.064293 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 02:28:16.064613 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:28:16.065966 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:16.159323 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1025
I20260620 02:28:16.177995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1159
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:16.193071 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:16.178838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.0.97.2:45967)
0620 02:28:16.179061 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:16.179068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:16.179089 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:28:16.179335 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:16.179339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:16.179406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:16.179518 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:16.180709 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.181502 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.182812 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.183192 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.183333 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:16.187604 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:16.187617 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:16.187633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:16.187673 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:16.190414 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.191091 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.191098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.191101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.191231 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.191600 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.191603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.191605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.191962 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:16.192225 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:16.192586 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 02:28:16.192751 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:28:16.193791 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:16.282605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1159
I20260620 02:28:16.300403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1293
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:16.316818 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:16.301365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.0.97.3:41171)
0620 02:28:16.301538 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:16.301543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:16.301709 (+ 166us) server_negotiation.cc:407] Connection header received
0620 02:28:16.301852 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:16.301857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:16.301937 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:16.302045 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:16.303339 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.304053 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.305507 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:16.305918 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:16.306027 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:16.310534 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:16.310550 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:16.310567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:16.310603 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:16.313702 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.314476 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.314489 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.314492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.314651 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:16.315084 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:16.315094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:16.315096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:16.315492 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:16.315814 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:16.316176 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:28:16.316418 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:28:16.317559 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:16.406250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1293
I20260620 02:28:16.423099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 932
I20260620 02:28:16.424299 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:16.532895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 932
2026-06-20T02:28:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4704 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[1469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[1469](info): set up 2 sockets
Jun 20 02:28:16 dist-test-slave-r706 krb5kdc[1469](info): commencing operation
krb5kdc: starting...
W20260620 02:28:17.633397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:17 dist-test-slave-r706 krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:17.833920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40619
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:18.171200 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:18.171519 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:18.171667 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:18.179184 1482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:18.179306 1482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:28:18.179378 1482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:18.179474 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:18.179543 1482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:18.179643 1482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:18.190303 1482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40619
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:18.191766 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:18.193289 1482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:18.205677 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:18.205729 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:18.205705 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:18.206593 1482 server_base.cc:1034] running on GCE node
I20260620 02:28:18.207388 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:18.208817 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:18.210073 1482 hybrid_clock.cc:648] HybridClock initialized: now 1781922498210032 us; error 73 us; skew 500 ppm
I20260620 02:28:18.210453 1482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:18 dist-test-slave-r706 krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:18.213876 1482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:18.215595 1482 webserver.cc:460] Webserver started at http://127.0.97.62:40121/ using document root <none> and password file <none>
I20260620 02:28:18.216318 1482 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:18.216413 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:18.216693 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:18.219902 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "3a386e00add740cb9eaa9854f5a5cb69"
format_stamp: "Formatted at 2026-06-20 02:28:18 on dist-test-slave-r706"
server_key: "8885f452168c14441d4d64ebce352a14"
server_key_iv: "2b935c2ddbf084dcde3782cb440040f1"
server_key_version: "encryptionkey@0"
I20260620 02:28:18.220731 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a386e00add740cb9eaa9854f5a5cb69"
format_stamp: "Formatted at 2026-06-20 02:28:18 on dist-test-slave-r706"
server_key: "8885f452168c14441d4d64ebce352a14"
server_key_iv: "2b935c2ddbf084dcde3782cb440040f1"
server_key_version: "encryptionkey@0"
I20260620 02:28:18.226394 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:28:18.230228 1497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:18.231799 1482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:28:18.232036 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "3a386e00add740cb9eaa9854f5a5cb69"
format_stamp: "Formatted at 2026-06-20 02:28:18 on dist-test-slave-r706"
server_key: "8885f452168c14441d4d64ebce352a14"
server_key_iv: "2b935c2ddbf084dcde3782cb440040f1"
server_key_version: "encryptionkey@0"
I20260620 02:28:18.232300 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:18.255227 1482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:18.259650 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:18.259967 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:18.283075 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40619
I20260620 02:28:18.283236 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40619 every 8 connection(s)
I20260620 02:28:18.284585 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:18.288658 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:18.293849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1482
I20260620 02:28:18.294077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:18.294905 388 external_mini_cluster.cc:1428] Setting key a2afde783ca63e6e37674ec1e41f003e
I20260620 02:28:18.300792 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: Bootstrap starting.
I20260620 02:28:18.304764 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:18.306102 1559 log.cc:824] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:18.309928 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: No bootstrap required, opened a new log
Jun 20 02:28:18 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:18.318387 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a386e00add740cb9eaa9854f5a5cb69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 02:28:18.318917 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:18.319073 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a386e00add740cb9eaa9854f5a5cb69, State: Initialized, Role: FOLLOWER
I20260620 02:28:18.319715 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a386e00add740cb9eaa9854f5a5cb69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 02:28:18.320003 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:18.320168 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:18.320375 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:18.320639 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:18.297212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.0.97.62:40619)
0620 02:28:18.298214 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:18.298229 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:18.298279 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:28:18.300020 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:18.300054 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:18.300471 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:18.300924 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:18.302844 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.304209 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.306099 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.307274 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.307500 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:18.312510 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:18.312521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:18.312537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:18.312608 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:18.315934 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:18.316715 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:18.316723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:18.316729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:18.316915 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:18.317360 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:18.317364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:18.317366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:18.317954 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:18.318301 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:18.318753 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 02:28:18.319263 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":500,"threads_started":1}
I20260620 02:28:18.322947 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a386e00add740cb9eaa9854f5a5cb69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 02:28:18.323483 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a386e00add740cb9eaa9854f5a5cb69; no voters:
I20260620 02:28:18.324097 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:18.324266 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:18.325106 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [term 1 LEADER]: Becoming Leader. State: Replica: 3a386e00add740cb9eaa9854f5a5cb69, State: Running, Role: LEADER
I20260620 02:28:18.325739 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a386e00add740cb9eaa9854f5a5cb69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 02:28:18.326335 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:18.328668 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a386e00add740cb9eaa9854f5a5cb69. Latest consensus state: current_term: 1 leader_uuid: "3a386e00add740cb9eaa9854f5a5cb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a386e00add740cb9eaa9854f5a5cb69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } } }
I20260620 02:28:18.329031 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:18.329952 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:18.329808 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a386e00add740cb9eaa9854f5a5cb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a386e00add740cb9eaa9854f5a5cb69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } } }
I20260620 02:28:18.330129 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:18.335469 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:18.345515 1569 catalog_manager.cc:1347] Generated new cluster ID: 2d8fd9da5c4848d3a93b586067dd8faa
I20260620 02:28:18.345670 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:18.356658 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:18.358476 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:18.369894 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: Generated new TSK 0
I20260620 02:28:18.370844 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:18.421166 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40619
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:18.765959 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:18.766271 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:18.766371 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:18.773766 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:18.773890 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:18.773974 1585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:28:18.774060 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:18.784394 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:18.785763 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:18.787091 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:18.800602 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:18.800803 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:18.800866 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:18.802241 1585 server_base.cc:1034] running on GCE node
I20260620 02:28:18.802932 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:18.803970 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:18.805240 1585 hybrid_clock.cc:648] HybridClock initialized: now 1781922498805178 us; error 43 us; skew 500 ppm
I20260620 02:28:18.805589 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:18 dist-test-slave-r706 krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:18.809072 1585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:18.810802 1585 webserver.cc:460] Webserver started at http://127.0.97.1:36789/ using document root <none> and password file <none>
I20260620 02:28:18.811578 1585 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:18.811719 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:18.812063 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:18.815237 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "7651c8d9f63643e5a0fd825cdac8f6e6"
format_stamp: "Formatted at 2026-06-20 02:28:18 on dist-test-slave-r706"
server_key: "b5fabaa1bb30dd42485355eecf5125da"
server_key_iv: "3dff9c36bd12b94d5534a15cfa55dc35"
server_key_version: "encryptionkey@0"
I20260620 02:28:18.816120 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7651c8d9f63643e5a0fd825cdac8f6e6"
format_stamp: "Formatted at 2026-06-20 02:28:18 on dist-test-slave-r706"
server_key: "b5fabaa1bb30dd42485355eecf5125da"
server_key_iv: "3dff9c36bd12b94d5534a15cfa55dc35"
server_key_version: "encryptionkey@0"
I20260620 02:28:18.822171 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:28:18.825944 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:18.827720 1585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:18.827919 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "7651c8d9f63643e5a0fd825cdac8f6e6"
format_stamp: "Formatted at 2026-06-20 02:28:18 on dist-test-slave-r706"
server_key: "b5fabaa1bb30dd42485355eecf5125da"
server_key_iv: "3dff9c36bd12b94d5534a15cfa55dc35"
server_key_version: "encryptionkey@0"
I20260620 02:28:18.828153 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:18.853118 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:18.858282 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:18.858566 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:18.861644 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:18.861774 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:18.861936 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:18.862109 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 02:28:18 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:18.887028 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:18.862159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35043 (local address 127.0.97.62:40619)
0620 02:28:18.862714 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:18.862723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:18.864584 (+ 1861us) server_negotiation.cc:407] Connection header received
0620 02:28:18.866510 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:18.866516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:18.866616 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:18.866744 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:18.869455 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.870901 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.872889 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.873781 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.873981 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:18.878724 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:18.878737 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:18.878746 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:18.878813 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:18.883126 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:18.884300 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:18.884313 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:18.884316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:18.884445 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:18.884950 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:18.884966 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:18.884971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:18.885410 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:18.885581 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:18.886397 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 02:28:18.886646 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
I20260620 02:28:18.888105 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:18.862998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.97.1:35043)
0620 02:28:18.864056 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:18.864108 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:18.866189 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:18.867002 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:18.867010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:18.867908 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:18.869164 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:18.869212 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.871130 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.871134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:18.872716 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:18.872730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.874142 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.874150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:18.874399 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:18.875380 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:18.875412 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:18.878416 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:18.883379 (+ 4963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:18.883396 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:18.883417 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:18.884151 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:18.884567 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:18.884577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:18.884581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:18.884809 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:18.885556 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:18.885569 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:18.886002 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 02:28:18.886287 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 02:28:18.886878 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":443,"threads_started":1}
I20260620 02:28:18.915180 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:18.897193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40801 (local address 127.0.97.62:40619)
0620 02:28:18.897436 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:18.897443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:18.897794 (+ 351us) server_negotiation.cc:407] Connection header received
0620 02:28:18.898070 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:18.898076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:18.898178 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:18.898292 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:18.900167 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.901238 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.904105 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.904745 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.905097 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:18.908014 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:18.908019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:18.908023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:18.908057 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:18.911557 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:18.912791 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:18.912799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:18.912805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:18.913022 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:18.913544 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:18.913552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:18.913557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:18.913915 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:18.914082 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:18.914525 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 02:28:18.914784 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:28:18.915297 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:18.896313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.97.1:40801)
0620 02:28:18.897624 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:18.897656 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:18.897882 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:18.898377 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:18.898386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:18.899410 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:18.899929 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:18.899949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.901476 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.901483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:18.903923 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:18.903947 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:18.905411 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:18.905417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:18.905555 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:18.906484 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:18.906517 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:18.907758 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:18.911747 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:18.911755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:18.911761 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:18.912638 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:18.913166 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:18.913171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:18.913175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:18.913339 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:18.914056 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:18.914061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:18.914260 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 02:28:18.914414 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:28:18.914785 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":730,"threads_started":1}
I20260620 02:28:18.920382 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39703
I20260620 02:28:18.920435 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39703 every 8 connection(s)
I20260620 02:28:18.921761 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:18.923997 1721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40619
I20260620 02:28:18.924247 1721 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:18.924912 1721 heartbeater.cc:511] Master 127.0.97.62:40619 requested a full tablet report, sending...
I20260620 02:28:18.927096 1514 ts_manager.cc:194] Registered new tserver with Master: 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703)
I20260620 02:28:18.928582 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35043
I20260620 02:28:18.932173 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1585
I20260620 02:28:18.932399 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:18.933168 388 external_mini_cluster.cc:1428] Setting key 9fd0908b911af76862797fc4e57b0ff0
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:18.980808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40619
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:19.331213 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:19.331557 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:19.331734 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:19.339622 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:19.339787 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:19.339959 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:28:19.340116 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:19.350313 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:19.351682 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:19.353089 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:19.366791 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:19.366847 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:19.366822 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:19.367671 1725 server_base.cc:1034] running on GCE node
I20260620 02:28:19.368315 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:19.369429 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:19.370669 1725 hybrid_clock.cc:648] HybridClock initialized: now 1781922499370603 us; error 42 us; skew 500 ppm
I20260620 02:28:19.370966 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:19 dist-test-slave-r706 krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:19.374339 1725 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:19.375787 1725 webserver.cc:460] Webserver started at http://127.0.97.2:34599/ using document root <none> and password file <none>
I20260620 02:28:19.376490 1725 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:19.376580 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:19.376859 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:19.379909 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "3c6dee7ac5ae4a1b9434b66817507dc3"
format_stamp: "Formatted at 2026-06-20 02:28:19 on dist-test-slave-r706"
server_key: "13792ab4fcf4f7599c8affd6fd214ff3"
server_key_iv: "db06bef92ab87ecdf062731f45cae7c7"
server_key_version: "encryptionkey@0"
I20260620 02:28:19.380668 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6dee7ac5ae4a1b9434b66817507dc3"
format_stamp: "Formatted at 2026-06-20 02:28:19 on dist-test-slave-r706"
server_key: "13792ab4fcf4f7599c8affd6fd214ff3"
server_key_iv: "db06bef92ab87ecdf062731f45cae7c7"
server_key_version: "encryptionkey@0"
I20260620 02:28:19.386174 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 02:28:19.389825 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:19.391273 1725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:19.391423 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "3c6dee7ac5ae4a1b9434b66817507dc3"
format_stamp: "Formatted at 2026-06-20 02:28:19 on dist-test-slave-r706"
server_key: "13792ab4fcf4f7599c8affd6fd214ff3"
server_key_iv: "db06bef92ab87ecdf062731f45cae7c7"
server_key_version: "encryptionkey@0"
I20260620 02:28:19.391608 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:19.404947 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:19.409708 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:19.410010 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:19.412658 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:19.412772 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:19.412921 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:19.413028 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:28:19 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:19.442543 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.414177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.97.2:57065)
0620 02:28:19.415106 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:19.415157 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:19.417623 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:19.418671 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:19.418684 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:19.419607 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:19.420496 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.420515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.422003 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.422007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.424063 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.424079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.425270 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.425279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.425680 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.426782 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:19.426825 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:19.430184 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:19.435074 (+ 4890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.435088 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.435115 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.435953 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.436523 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.436531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.436541 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.437241 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.438096 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:19.438108 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:19.438814 (+ 706us) client_negotiation.cc:769] Sending connection context
0620 02:28:19.439247 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 02:28:19.440097 (+ 850us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":426,"threads_started":1}
I20260620 02:28:19.444612 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.413556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57065 (local address 127.0.97.62:40619)
0620 02:28:19.413759 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:19.413765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:19.415309 (+ 1544us) server_negotiation.cc:407] Connection header received
0620 02:28:19.418064 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:19.418075 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:19.418200 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:19.418323 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:19.420701 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.421812 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.424323 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.424947 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.425140 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.430507 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:19.430515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:19.430521 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:19.430574 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:19.434879 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.436140 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.436149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.436154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.436322 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.437436 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.437444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.437450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.437912 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:19.443712 (+ 5800us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:19.443902 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 02:28:19.444196 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:28:19.454222 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39565
I20260620 02:28:19.454324 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39565 every 8 connection(s)
I20260620 02:28:19.456048 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:19.458416 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40619
I20260620 02:28:19.458664 1860 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:19.459509 1860 heartbeater.cc:511] Master 127.0.97.62:40619 requested a full tablet report, sending...
I20260620 02:28:19.461066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1725
I20260620 02:28:19.461253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:19.461493 1514 ts_manager.cc:194] Registered new tserver with Master: 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565)
I20260620 02:28:19.461838 388 external_mini_cluster.cc:1428] Setting key 3953009ed6dedd73b6a0d5fcd70b65d9
I20260620 02:28:19.463076 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57065
I20260620 02:28:19.468281 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.452539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.97.2:46445)
0620 02:28:19.453977 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:19.454006 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:19.454206 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:19.454634 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:19.454639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:19.455232 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:19.455628 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.455644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.456819 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.456826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.458186 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.458198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.459232 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.459237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.459377 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.460082 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:19.460099 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:19.461225 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:19.465796 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.465800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.465803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.466208 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.466632 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.466636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.466638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.466760 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.467465 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:19.467468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:19.467618 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 02:28:19.467725 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:28:19.467961 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"thread_start_us":1043,"threads_started":1}
I20260620 02:28:19.468552 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.452551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46445 (local address 127.0.97.62:40619)
0620 02:28:19.452806 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:19.452814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:19.454142 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 02:28:19.454367 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:19.454372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:19.454475 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:19.454584 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:19.455812 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.456636 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.458367 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.459053 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.459197 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.461456 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:19.461463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:19.461467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:19.461511 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:19.465639 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.466355 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.466364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.466370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.466522 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.466875 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.466882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.466888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.467342 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:19.467505 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:19.467888 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:28:19.468144 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:19.505208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40619
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:19.842710 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:19.843014 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:19.843178 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:19.850929 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:19.851063 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:19.851212 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:28:19.851343 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:19.861577 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:19.862926 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:19.864310 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:19.878893 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:19.878890 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:19.879365 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:19.879487 1864 server_base.cc:1034] running on GCE node
I20260620 02:28:19.880178 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:19.881235 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:19.882519 1864 hybrid_clock.cc:648] HybridClock initialized: now 1781922499882441 us; error 52 us; skew 500 ppm
I20260620 02:28:19.882928 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:19 dist-test-slave-r706 krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:19.886283 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:19.887884 1864 webserver.cc:460] Webserver started at http://127.0.97.3:43355/ using document root <none> and password file <none>
I20260620 02:28:19.888585 1864 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:19.888680 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:19.888962 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:19.892395 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "7bf9c61a75b14a598a1ef79dc60ca943"
format_stamp: "Formatted at 2026-06-20 02:28:19 on dist-test-slave-r706"
server_key: "c18867159418b72d82a773c44156e6c9"
server_key_iv: "10d7ea74b7c8eebc7e7cf81d5554bf58"
server_key_version: "encryptionkey@0"
I20260620 02:28:19.893329 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7bf9c61a75b14a598a1ef79dc60ca943"
format_stamp: "Formatted at 2026-06-20 02:28:19 on dist-test-slave-r706"
server_key: "c18867159418b72d82a773c44156e6c9"
server_key_iv: "10d7ea74b7c8eebc7e7cf81d5554bf58"
server_key_version: "encryptionkey@0"
I20260620 02:28:19.899219 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:19.903504 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:19.905340 1864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 02:28:19.905555 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "7bf9c61a75b14a598a1ef79dc60ca943"
format_stamp: "Formatted at 2026-06-20 02:28:19 on dist-test-slave-r706"
server_key: "c18867159418b72d82a773c44156e6c9"
server_key_iv: "10d7ea74b7c8eebc7e7cf81d5554bf58"
server_key_version: "encryptionkey@0"
I20260620 02:28:19.905802 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:19.929639 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:19.931492 1721 heartbeater.cc:503] Master 127.0.97.62:40619 was elected leader, sending a full tablet report...
I20260620 02:28:19.934844 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:19.935164 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:19.938309 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:19.938434 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:19.938594 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:19.938674 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:28:19 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:19.964573 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.939166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37835 (local address 127.0.97.62:40619)
0620 02:28:19.939372 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:19.939378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:19.941019 (+ 1641us) server_negotiation.cc:407] Connection header received
0620 02:28:19.943360 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:19.943372 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:19.943451 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:19.943571 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:19.946530 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.947960 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.950439 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.951075 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.951233 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.956748 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:19.956759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:19.956768 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:19.956821 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:19.960627 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.961947 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.961955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.961960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.962151 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.962666 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.962671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.962675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.962981 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:19.963154 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:19.963930 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 02:28:19.964216 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:28:19.965992 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.939799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.97.3:37835)
0620 02:28:19.940794 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:19.940844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:19.942954 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:19.944246 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:19.944254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:19.945217 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:19.946227 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.946288 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.948194 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.948199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.950204 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.950226 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.951383 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.951393 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.951679 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.952619 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:19.952707 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:19.956384 (+ 3677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:19.960806 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.960823 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.960851 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.961785 (+ 934us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.962282 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.962288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.962295 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.962533 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.963130 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:19.963138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:19.963545 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 02:28:19.963826 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 02:28:19.964398 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":432,"threads_started":1}
I20260620 02:28:19.988950 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40083
I20260620 02:28:19.988999 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40083 every 8 connection(s)
I20260620 02:28:19.990990 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:19.991811 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.972928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56667 (local address 127.0.97.62:40619)
0620 02:28:19.973232 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:19.973242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:19.974525 (+ 1283us) server_negotiation.cc:407] Connection header received
0620 02:28:19.974918 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:19.974927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:19.975038 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:19.975123 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:19.976783 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.977771 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.979976 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.980933 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.981140 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.984549 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:19.984563 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:19.984567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:19.984605 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:19.987859 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.989086 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.989095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.989101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.989277 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:19.989740 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:19.989744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:19.989746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:19.990207 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:19.990555 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:19.991170 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 02:28:19.991385 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 02:28:19.991995 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:19.972774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.97.3:56667)
0620 02:28:19.974352 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:19.974389 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:19.974711 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:19.975368 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:19.975375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:19.976088 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:19.976614 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.976627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.977989 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.977994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.979703 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:19.979720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:19.981480 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:19.981491 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:19.981756 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:19.982669 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:19.982703 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:19.984288 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:19.988079 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.988089 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.988093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.988893 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.989419 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:19.989424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:19.989428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:19.989579 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:19.990522 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:19.990529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:19.990751 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 02:28:19.990935 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 02:28:19.991328 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":1230,"threads_started":1}
I20260620 02:28:19.993606 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40619
I20260620 02:28:19.993898 1999 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:19.994860 1999 heartbeater.cc:511] Master 127.0.97.62:40619 requested a full tablet report, sending...
I20260620 02:28:19.995406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 1864
I20260620 02:28:19.995625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:19.996323 388 external_mini_cluster.cc:1428] Setting key eba24d3fbe329d07a88d59ee6b7ccce3
I20260620 02:28:19.997184 1513 ts_manager.cc:194] Registered new tserver with Master: 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083)
I20260620 02:28:19.998344 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37835
I20260620 02:28:20.000056 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:28:20 dist-test-slave-r706 krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:28:20 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:20.049737 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.034118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.0.97.62:40619)
0620 02:28:20.034351 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.034358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.034507 (+ 149us) server_negotiation.cc:407] Connection header received
0620 02:28:20.034658 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.034662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.034729 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.034820 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:20.036627 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.037644 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.039045 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.039459 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.039568 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.043828 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:20.043833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:20.043837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:20.043878 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:20.046805 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:20.047571 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:20.047584 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:20.047588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:20.047730 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:20.048150 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:20.048156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:20.048159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:20.048441 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:20.048700 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.049086 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.049349 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:28:20.054214 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:20.061292 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:20.097354 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.074142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39703 (local address 127.0.0.1:52636)
0620 02:28:20.086438 (+ 12296us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.086521 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.086863 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.088324 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.088338 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.088384 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.089044 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.089064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.091404 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.091409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.094114 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.094133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.096304 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.096312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.096540 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.096617 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.096748 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.096957 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11895,"thread_start_us":13033,"threads_started":1}
I20260620 02:28:20.097388 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.074591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.97.1:39703)
0620 02:28:20.085748 (+ 11157us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.085761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.087181 (+ 1420us) server_negotiation.cc:407] Connection header received
0620 02:28:20.087341 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.087353 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.087899 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.089358 (+ 1459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.089640 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.091255 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.094334 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.095952 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.096084 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.096296 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.096859 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.097068 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10899,"thread_start_us":10707,"threads_started":1}
I20260620 02:28:20.099133 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.087743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39565 (local address 127.0.0.1:43332)
0620 02:28:20.090178 (+ 2435us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.090197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.090306 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.091409 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.091413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.091441 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.092047 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.092063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.094169 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.094177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.096378 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.096394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.098183 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.098191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.098418 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.098457 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.098528 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.098656 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2316,"spinlock_wait_cycles":2432,"thread_start_us":2574,"threads_started":1}
I20260620 02:28:20.099296 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.074148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40083 (local address 127.0.0.1:45708)
0620 02:28:20.086967 (+ 12819us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.086989 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.087122 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.091490 (+ 4368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.091495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.091540 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.092074 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.092085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.093757 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.093764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.095807 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.095821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.098455 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.098461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.098616 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.098654 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.098715 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.098847 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12682,"mutex_wait_us":51,"spinlock_wait_cycles":3200,"thread_start_us":11617,"threads_started":1}
I20260620 02:28:20.100016 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.087324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.0.97.2:39565)
0620 02:28:20.090841 (+ 3517us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.090850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.090882 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:28:20.090969 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.090976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.091183 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.092261 (+ 1078us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.092487 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.094005 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.096556 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.097901 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.099268 (+ 1367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.099531 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.099602 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.099794 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3309,"thread_start_us":3194,"threads_started":1}
I20260620 02:28:20.100374 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.074341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.97.3:40083)
0620 02:28:20.090055 (+ 15714us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.090069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.090115 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:28:20.090238 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.090250 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.091022 (+ 772us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.091233 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.092379 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.093586 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.096005 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.097842 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.099525 (+ 1683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.099766 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.099840 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.100030 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15467,"thread_start_us":15239,"threads_started":1}
I20260620 02:28:20.102941 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 3f16eb5b34b44a29a7e82908b5ffd34a (DEFAULT_TABLE table=test-table [id=22d56b911434424f9cbff5313fd747f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:20.104214 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 3f16eb5b34b44a29a7e82908b5ffd34a (DEFAULT_TABLE table=test-table [id=22d56b911434424f9cbff5313fd747f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:20.104941 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f16eb5b34b44a29a7e82908b5ffd34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.105057 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 3f16eb5b34b44a29a7e82908b5ffd34a (DEFAULT_TABLE table=test-table [id=22d56b911434424f9cbff5313fd747f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:20.105700 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f16eb5b34b44a29a7e82908b5ffd34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.106395 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f16eb5b34b44a29a7e82908b5ffd34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.118248 2018 tablet_bootstrap.cc:492] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Bootstrap starting.
I20260620 02:28:20.120428 2019 tablet_bootstrap.cc:492] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Bootstrap starting.
I20260620 02:28:20.123153 2020 tablet_bootstrap.cc:492] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Bootstrap starting.
I20260620 02:28:20.124977 2019 tablet_bootstrap.cc:654] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.124969 2018 tablet_bootstrap.cc:654] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.126121 2020 tablet_bootstrap.cc:654] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.126557 2019 log.cc:824] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:20.126554 2018 log.cc:824] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:20.127736 2020 log.cc:824] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:20.129874 2019 tablet_bootstrap.cc:492] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: No bootstrap required, opened a new log
I20260620 02:28:20.130012 2018 tablet_bootstrap.cc:492] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: No bootstrap required, opened a new log
I20260620 02:28:20.130239 2019 ts_tablet_manager.cc:1397] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 02:28:20.130297 2018 ts_tablet_manager.cc:1397] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 02:28:20.130846 2020 tablet_bootstrap.cc:492] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: No bootstrap required, opened a new log
I20260620 02:28:20.131167 2020 ts_tablet_manager.cc:1397] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:28:20.140456 2019 raft_consensus.cc:348] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.140983 2019 raft_consensus.cc:374] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.141175 2019 raft_consensus.cc:729] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7651c8d9f63643e5a0fd825cdac8f6e6, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.141909 2019 consensus_queue.cc:260] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.143836 2020 raft_consensus.cc:348] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.144508 2020 raft_consensus.cc:374] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.144724 2020 raft_consensus.cc:729] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6dee7ac5ae4a1b9434b66817507dc3, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.145506 2019 ts_tablet_manager.cc:1428] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260620 02:28:20.145171 2018 raft_consensus.cc:348] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.145807 2018 raft_consensus.cc:374] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.146059 2018 raft_consensus.cc:729] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf9c61a75b14a598a1ef79dc60ca943, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.145867 2020 consensus_queue.cc:260] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.147114 2018 consensus_queue.cc:260] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.153136 1860 heartbeater.cc:503] Master 127.0.97.62:40619 was elected leader, sending a full tablet report...
I20260620 02:28:20.153739 1999 heartbeater.cc:503] Master 127.0.97.62:40619 was elected leader, sending a full tablet report...
I20260620 02:28:20.154137 2020 ts_tablet_manager.cc:1428] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Time spent starting tablet: real 0.023s user 0.016s sys 0.005s
I20260620 02:28:20.154484 2018 ts_tablet_manager.cc:1428] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Time spent starting tablet: real 0.024s user 0.012s sys 0.009s
W20260620 02:28:20.174263 1722 tablet.cc:2369] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:20.208364 1861 tablet.cc:2369] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:20.249152 2000 tablet.cc:2369] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:20.298018 2026 raft_consensus.cc:482] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:20.298560 2026 raft_consensus.cc:504] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.300621 2026 leader_election.cc:288] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703), 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565)
I20260620 02:28:20.310885 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.301379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51119 (local address 127.0.97.1:39703)
0620 02:28:20.301643 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.301649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.302008 (+ 359us) server_negotiation.cc:407] Connection header received
0620 02:28:20.302260 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.302270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.302423 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.302568 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.303458 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.304679 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.308044 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.309693 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.309874 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.310027 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.310439 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.310589 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:28:20.311146 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.301539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39703 (local address 127.0.97.3:51119)
0620 02:28:20.301871 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.301894 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.302057 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.302632 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.302637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.302667 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.303010 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.303024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.304853 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.304857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.307805 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.307834 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.309959 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.309965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.310200 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.310252 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.310418 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.310652 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":5}
I20260620 02:28:20.311568 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.301360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50935 (local address 127.0.97.2:39565)
0620 02:28:20.301644 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.301650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.302968 (+ 1318us) server_negotiation.cc:407] Connection header received
0620 02:28:20.303146 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.303150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.303209 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.303281 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.304025 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.305238 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.308451 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.310149 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.310269 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.310405 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.311130 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.311279 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:28:20.311599 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.301641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39565 (local address 127.0.97.3:50935)
0620 02:28:20.302819 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.302838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.303026 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.303314 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.303317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.303336 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.303827 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.303835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.305415 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.305421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.308232 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.308245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.310474 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.310480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.310715 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.310748 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.310977 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.311160 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":745,"threads_started":1}
I20260620 02:28:20.312862 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f16eb5b34b44a29a7e82908b5ffd34a" candidate_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" is_pre_election: true
I20260620 02:28:20.312871 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f16eb5b34b44a29a7e82908b5ffd34a" candidate_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" is_pre_election: true
I20260620 02:28:20.313570 1810 raft_consensus.cc:2454] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf9c61a75b14a598a1ef79dc60ca943 in term 0.
I20260620 02:28:20.313567 1676 raft_consensus.cc:2454] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf9c61a75b14a598a1ef79dc60ca943 in term 0.
I20260620 02:28:20.314390 1881 leader_election.cc:302] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6dee7ac5ae4a1b9434b66817507dc3, 7bf9c61a75b14a598a1ef79dc60ca943; no voters:
I20260620 02:28:20.314919 2026 raft_consensus.cc:2781] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:20.315126 2026 raft_consensus.cc:482] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:20.315232 2026 raft_consensus.cc:3037] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:20.318172 2026 raft_consensus.cc:504] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.318990 2026 leader_election.cc:288] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [CANDIDATE]: Term 1 election: Requested vote from peers 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703), 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565)
I20260620 02:28:20.319490 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f16eb5b34b44a29a7e82908b5ffd34a" candidate_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6"
I20260620 02:28:20.319638 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f16eb5b34b44a29a7e82908b5ffd34a" candidate_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3"
I20260620 02:28:20.319752 1676 raft_consensus.cc:3037] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:20.319957 1810 raft_consensus.cc:3037] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:20.323014 1810 raft_consensus.cc:2454] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf9c61a75b14a598a1ef79dc60ca943 in term 1.
I20260620 02:28:20.323179 1676 raft_consensus.cc:2454] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf9c61a75b14a598a1ef79dc60ca943 in term 1.
I20260620 02:28:20.323598 1881 leader_election.cc:302] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6dee7ac5ae4a1b9434b66817507dc3, 7bf9c61a75b14a598a1ef79dc60ca943; no voters:
I20260620 02:28:20.324033 2026 raft_consensus.cc:2781] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:20.324672 2026 raft_consensus.cc:686] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf9c61a75b14a598a1ef79dc60ca943, State: Running, Role: LEADER
I20260620 02:28:20.325485 2026 consensus_queue.cc:237] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } }
I20260620 02:28:20.331087 1514 catalog_manager.cc:5495] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } health_report { overall_health: UNKNOWN } } }
I20260620 02:28:20.413098 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.406139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.97.3:40083)
0620 02:28:20.406340 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.406348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.406655 (+ 307us) server_negotiation.cc:407] Connection header received
0620 02:28:20.407019 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.407023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.407108 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.407267 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:20.407696 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.408517 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.410288 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.410870 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.411025 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.411722 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:20.412317 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:20.412517 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.412623 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.412715 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:28:20.423612 1676 raft_consensus.cc:1261] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Refusing update from remote peer 7bf9c61a75b14a598a1ef79dc60ca943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:20.423612 1810 raft_consensus.cc:1261] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Refusing update from remote peer 7bf9c61a75b14a598a1ef79dc60ca943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:20.425096 2028 consensus_queue.cc:1035] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:20.425732 2026 consensus_queue.cc:1035] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:20.441493 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298754561721491456 to 7298754561338236928. Current Snapshot: MvccSnapshot[applied={T|T < 7298754561721491456}]
I20260620 02:28:20.442409 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298754561721491456 to 7298754561338236928. Current Snapshot: MvccSnapshot[applied={T|T < 7298754561721491456}]
I20260620 02:28:20.450330 2033 mvcc.cc:205] Tried to move back new op lower bound from 7298754561721491456 to 7298754561338236928. Current Snapshot: MvccSnapshot[applied={T|T < 7298754561721491456}]
I20260620 02:28:20.470703 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:20.471244 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:20.476909 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: Sending DeleteTablet for 3 replicas of tablet 3f16eb5b34b44a29a7e82908b5ffd34a
I20260620 02:28:20.478632 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f16eb5b34b44a29a7e82908b5ffd34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45708
I20260620 02:28:20.478724 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f16eb5b34b44a29a7e82908b5ffd34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52636
I20260620 02:28:20.479274 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f16eb5b34b44a29a7e82908b5ffd34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43332
I20260620 02:28:20.481423 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:20.482528 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:20.488749 2040 tablet_replica.cc:330] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: stopping tablet replica
I20260620 02:28:20.489432 2040 raft_consensus.cc:2229] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:20.490654 2040 raft_consensus.cc:2258] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:20.491992 2042 tablet_replica.cc:330] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: stopping tablet replica
I20260620 02:28:20.492151 2041 tablet_replica.cc:330] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: stopping tablet replica
I20260620 02:28:20.492633 2041 raft_consensus.cc:2229] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:20.492633 2042 raft_consensus.cc:2229] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:20.493449 2041 raft_consensus.cc:2258] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:20.493449 2042 raft_consensus.cc:2258] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:20.493784 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 1069ea39090d46fba9a8337bd4db9479 (DEFAULT_TABLE table=test-table [id=08b359ec9c284bc8b439b4be248749c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:20.494928 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1069ea39090d46fba9a8337bd4db9479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.494745 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 1069ea39090d46fba9a8337bd4db9479 (DEFAULT_TABLE table=test-table [id=08b359ec9c284bc8b439b4be248749c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:20.495471 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1069ea39090d46fba9a8337bd4db9479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.495913 2041 ts_tablet_manager.cc:1905] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:20.497037 2042 ts_tablet_manager.cc:1905] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:20.499295 2040 ts_tablet_manager.cc:1905] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:20.506233 2042 ts_tablet_manager.cc:1918] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:20.506608 2042 log.cc:1196] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/3f16eb5b34b44a29a7e82908b5ffd34a
I20260620 02:28:20.507642 2020 tablet_bootstrap.cc:492] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Bootstrap starting.
I20260620 02:28:20.502310 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 1069ea39090d46fba9a8337bd4db9479 (DEFAULT_TABLE table=test-table [id=08b359ec9c284bc8b439b4be248749c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:20.508980 2041 ts_tablet_manager.cc:1918] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:20.509356 2041 log.cc:1196] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/3f16eb5b34b44a29a7e82908b5ffd34a
I20260620 02:28:20.509956 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1069ea39090d46fba9a8337bd4db9479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.510249 2041 ts_tablet_manager.cc:1939] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7651c8d9f63643e5a0fd825cdac8f6e6: Deleting consensus metadata
I20260620 02:28:20.510358 2020 tablet_bootstrap.cc:654] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.513679 2042 ts_tablet_manager.cc:1939] T 3f16eb5b34b44a29a7e82908b5ffd34a P 3c6dee7ac5ae4a1b9434b66817507dc3: Deleting consensus metadata
I20260620 02:28:20.513818 1500 catalog_manager.cc:4841] TS 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703): tablet 3f16eb5b34b44a29a7e82908b5ffd34a (table test-table [id=22d56b911434424f9cbff5313fd747f0]) successfully deleted
I20260620 02:28:20.516573 1499 catalog_manager.cc:4841] TS 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565): tablet 3f16eb5b34b44a29a7e82908b5ffd34a (table test-table [id=22d56b911434424f9cbff5313fd747f0]) successfully deleted
I20260620 02:28:20.518543 2020 tablet_bootstrap.cc:492] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: No bootstrap required, opened a new log
I20260620 02:28:20.518764 2020 ts_tablet_manager.cc:1397] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 02:28:20.519724 2020 raft_consensus.cc:348] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.520157 2020 raft_consensus.cc:374] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.520390 2020 raft_consensus.cc:729] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6dee7ac5ae4a1b9434b66817507dc3, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.520901 2020 consensus_queue.cc:260] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.522071 2020 ts_tablet_manager.cc:1428] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 02:28:20.524036 2018 tablet_bootstrap.cc:492] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Bootstrap starting.
I20260620 02:28:20.525246 2019 tablet_bootstrap.cc:492] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Bootstrap starting.
I20260620 02:28:20.527243 2040 ts_tablet_manager.cc:1918] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:20.527585 2040 log.cc:1196] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/3f16eb5b34b44a29a7e82908b5ffd34a
I20260620 02:28:20.527832 2019 tablet_bootstrap.cc:654] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.528005 2018 tablet_bootstrap.cc:654] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.528761 2040 ts_tablet_manager.cc:1939] T 3f16eb5b34b44a29a7e82908b5ffd34a P 7bf9c61a75b14a598a1ef79dc60ca943: Deleting consensus metadata
I20260620 02:28:20.531754 1499 catalog_manager.cc:4841] TS 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083): tablet 3f16eb5b34b44a29a7e82908b5ffd34a (table test-table [id=22d56b911434424f9cbff5313fd747f0]) successfully deleted
I20260620 02:28:20.539362 2019 tablet_bootstrap.cc:492] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: No bootstrap required, opened a new log
I20260620 02:28:20.539592 2019 ts_tablet_manager.cc:1397] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260620 02:28:20.540674 2019 raft_consensus.cc:348] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.540934 2019 raft_consensus.cc:374] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.541127 2019 raft_consensus.cc:729] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7651c8d9f63643e5a0fd825cdac8f6e6, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.541206 2018 tablet_bootstrap.cc:492] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: No bootstrap required, opened a new log
I20260620 02:28:20.541448 2018 ts_tablet_manager.cc:1397] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20260620 02:28:20.541628 2019 consensus_queue.cc:260] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.542675 2018 raft_consensus.cc:348] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.542963 2018 raft_consensus.cc:374] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.543229 2018 raft_consensus.cc:729] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf9c61a75b14a598a1ef79dc60ca943, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.543470 2019 ts_tablet_manager.cc:1428] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 02:28:20.543848 2018 consensus_queue.cc:260] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.545486 2018 ts_tablet_manager.cc:1428] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:28:20.652618 2025 raft_consensus.cc:482] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:20.653148 2025 raft_consensus.cc:504] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.655335 2025 leader_election.cc:288] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703), 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083)
I20260620 02:28:20.667737 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.655824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38055 (local address 127.0.97.3:40083)
0620 02:28:20.656180 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.656191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.657961 (+ 1770us) server_negotiation.cc:407] Connection header received
0620 02:28:20.658203 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.658209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.658328 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.658482 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.659872 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.661449 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.664155 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.665759 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.665969 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.666130 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.666989 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.667190 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260620 02:28:20.668334 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.655855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40083 (local address 127.0.97.2:38055)
0620 02:28:20.657809 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.657827 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.658006 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.658506 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.658511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.658555 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.659573 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.659587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.661604 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.661610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.663924 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.663940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.666422 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.666431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.666759 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.666811 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.667564 (+ 753us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.667735 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"mutex_wait_us":29,"thread_start_us":1470,"threads_started":1}
I20260620 02:28:20.668447 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.656433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47519 (local address 127.0.97.1:39703)
0620 02:28:20.656649 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.656655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.657329 (+ 674us) server_negotiation.cc:407] Connection header received
0620 02:28:20.657751 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.657757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.657867 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.657978 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:20.659450 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.661137 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.664849 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.666768 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.666928 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.667076 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.667849 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.668018 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:28:20.669551 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.655727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39703 (local address 127.0.97.2:47519)
0620 02:28:20.657146 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.657178 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.657493 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.658289 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.658294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.658322 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:20.659228 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.659242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.661300 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.661306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.664606 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.664622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.667330 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.667338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.667568 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.667621 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.668748 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.668909 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":102,"thread_start_us":904,"threads_started":1}
I20260620 02:28:20.669714 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1069ea39090d46fba9a8337bd4db9479" candidate_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" is_pre_election: true
I20260620 02:28:20.670351 1949 raft_consensus.cc:2454] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6dee7ac5ae4a1b9434b66817507dc3 in term 0.
I20260620 02:28:20.670702 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1069ea39090d46fba9a8337bd4db9479" candidate_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" is_pre_election: true
I20260620 02:28:20.671139 1675 raft_consensus.cc:2454] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6dee7ac5ae4a1b9434b66817507dc3 in term 0.
I20260620 02:28:20.671409 1742 leader_election.cc:302] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6dee7ac5ae4a1b9434b66817507dc3, 7bf9c61a75b14a598a1ef79dc60ca943; no voters:
I20260620 02:28:20.672051 2025 raft_consensus.cc:2781] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:20.672273 2025 raft_consensus.cc:482] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:20.672462 2025 raft_consensus.cc:3037] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:20.677067 2025 raft_consensus.cc:504] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.678201 2025 leader_election.cc:288] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [CANDIDATE]: Term 1 election: Requested vote from peers 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703), 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083)
I20260620 02:28:20.678859 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1069ea39090d46fba9a8337bd4db9479" candidate_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf9c61a75b14a598a1ef79dc60ca943"
I20260620 02:28:20.678901 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1069ea39090d46fba9a8337bd4db9479" candidate_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6"
I20260620 02:28:20.679167 1949 raft_consensus.cc:3037] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:20.679219 1675 raft_consensus.cc:3037] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:20.682788 1949 raft_consensus.cc:2454] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6dee7ac5ae4a1b9434b66817507dc3 in term 1.
I20260620 02:28:20.683355 1742 leader_election.cc:302] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6dee7ac5ae4a1b9434b66817507dc3, 7bf9c61a75b14a598a1ef79dc60ca943; no voters:
I20260620 02:28:20.683723 2025 raft_consensus.cc:2781] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:20.684602 2025 raft_consensus.cc:686] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6dee7ac5ae4a1b9434b66817507dc3, State: Running, Role: LEADER
I20260620 02:28:20.685163 2025 consensus_queue.cc:237] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.685547 1675 raft_consensus.cc:2454] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6dee7ac5ae4a1b9434b66817507dc3 in term 1.
I20260620 02:28:20.689967 1514 catalog_manager.cc:5495] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:28:20 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:20.756764 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.736266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.97.62:40619)
0620 02:28:20.737491 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:20.737497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:20.737971 (+ 474us) server_negotiation.cc:407] Connection header received
0620 02:28:20.738222 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:20.738226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:20.738286 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:20.738388 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:20.740490 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.741600 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.744074 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.744715 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.744877 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.749933 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:20.749939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:20.749943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:20.749973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:20.753407 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:20.754272 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:20.754281 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:20.754284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:20.754441 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:20.754884 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:20.754888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:20.754890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:20.755188 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:20.755344 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:20.755848 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 02:28:20.756199 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":967,"thread_start_us":663,"threads_started":1}
I20260620 02:28:20.756712 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:20.736314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40619 (local address 127.0.0.1:54344)
0620 02:28:20.737833 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:20.737855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:20.738053 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:20.738490 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:20.738496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:20.739748 (+ 1252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:20.740248 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.740262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.741783 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.741789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.743903 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:20.743917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:20.744962 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:20.744976 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:20.745264 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:20.746232 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:20.746256 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:20.749637 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:20.753544 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:20.753560 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:20.753568 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:20.754149 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:20.754548 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:20.754555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:20.754559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:20.754751 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:20.755319 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:20.755327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:20.755527 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 02:28:20.755746 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 02:28:20.756064 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":1102,"threads_started":1}
I20260620 02:28:20.764145 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54344:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:28:20.765916 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:20.775105 1651 tablet_service.cc:1467] Processing CreateTablet for tablet b0006f199d7f43e0a1fbc99642b0ee00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a74641ff48cf4676b2f8411630fa578c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:28:20.775421 1789 tablet_service.cc:1467] Processing CreateTablet for tablet b0006f199d7f43e0a1fbc99642b0ee00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a74641ff48cf4676b2f8411630fa578c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:28:20.775867 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0006f199d7f43e0a1fbc99642b0ee00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.776055 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0006f199d7f43e0a1fbc99642b0ee00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.783644 2019 tablet_bootstrap.cc:492] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: Bootstrap starting.
I20260620 02:28:20.785687 2019 tablet_bootstrap.cc:654] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.785794 1929 tablet_service.cc:1467] Processing CreateTablet for tablet b0006f199d7f43e0a1fbc99642b0ee00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a74641ff48cf4676b2f8411630fa578c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:28:20.786749 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0006f199d7f43e0a1fbc99642b0ee00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:20.789237 2019 tablet_bootstrap.cc:492] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: No bootstrap required, opened a new log
I20260620 02:28:20.789489 2019 ts_tablet_manager.cc:1397] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 02:28:20.789770 2020 tablet_bootstrap.cc:492] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: Bootstrap starting.
I20260620 02:28:20.790611 2019 raft_consensus.cc:348] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.790952 2019 raft_consensus.cc:374] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.791087 2019 raft_consensus.cc:729] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7651c8d9f63643e5a0fd825cdac8f6e6, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.791474 2019 consensus_queue.cc:260] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.792320 2019 ts_tablet_manager.cc:1428] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:20.792519 2020 tablet_bootstrap.cc:654] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.792241 2024 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:20.792945 2024 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:20.801201 2018 tablet_bootstrap.cc:492] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: Bootstrap starting.
I20260620 02:28:20.803738 2020 tablet_bootstrap.cc:492] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: No bootstrap required, opened a new log
I20260620 02:28:20.804036 2020 ts_tablet_manager.cc:1397] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 02:28:20.804440 2018 tablet_bootstrap.cc:654] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:20.804945 2020 raft_consensus.cc:348] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.805346 2020 raft_consensus.cc:374] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.805465 2020 raft_consensus.cc:729] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6dee7ac5ae4a1b9434b66817507dc3, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.805868 2020 consensus_queue.cc:260] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.806398 2025 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:20.806689 2020 ts_tablet_manager.cc:1428] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:28:20.806957 2025 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:20.809333 2018 tablet_bootstrap.cc:492] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: No bootstrap required, opened a new log
I20260620 02:28:20.809923 2018 ts_tablet_manager.cc:1397] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260620 02:28:20.811875 2018 raft_consensus.cc:348] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.812333 2018 raft_consensus.cc:374] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:20.812505 2018 raft_consensus.cc:729] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf9c61a75b14a598a1ef79dc60ca943, State: Initialized, Role: FOLLOWER
I20260620 02:28:20.813059 2018 consensus_queue.cc:260] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:20.813639 2018 ts_tablet_manager.cc:1428] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:28:20.813614 2028 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:20.814035 2028 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:21.059604 2024 raft_consensus.cc:482] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:21.059988 2024 raft_consensus.cc:504] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:21.062609 2024 leader_election.cc:288] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565), 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083)
I20260620 02:28:21.073559 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.062793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39565 (local address 127.0.97.1:34567)
0620 02:28:21.065788 (+ 2995us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:21.065809 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:21.066002 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:21.066664 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:21.066672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:21.066710 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:21.067130 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.067146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.069152 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.069157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.071279 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.071291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.072546 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.072553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.072724 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.072749 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:28:21.072862 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:28:21.073041 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2880,"thread_start_us":2779,"threads_started":1}
I20260620 02:28:21.073606 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.062795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34567 (local address 127.0.97.2:39565)
0620 02:28:21.065622 (+ 2827us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.065629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.065924 (+ 295us) server_negotiation.cc:407] Connection header received
0620 02:28:21.066209 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.066214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.066307 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.066432 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:21.067346 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.068955 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.071443 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.072216 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.072383 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.072545 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.072889 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.073288 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2680,"thread_start_us":2538,"threads_started":1}
I20260620 02:28:21.074563 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0006f199d7f43e0a1fbc99642b0ee00" candidate_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" is_pre_election: true
I20260620 02:28:21.074962 1809 raft_consensus.cc:2454] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7651c8d9f63643e5a0fd825cdac8f6e6 in term 0.
I20260620 02:28:21.075511 1602 leader_election.cc:302] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6dee7ac5ae4a1b9434b66817507dc3, 7651c8d9f63643e5a0fd825cdac8f6e6; no voters:
I20260620 02:28:21.075958 2024 raft_consensus.cc:2781] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:21.076104 2024 raft_consensus.cc:482] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:21.076206 2024 raft_consensus.cc:3037] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:21.077884 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.066163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40925 (local address 127.0.97.3:40083)
0620 02:28:21.066377 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.066383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.067790 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 02:28:21.067987 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.067993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.068099 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.068208 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:21.069041 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.070299 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.074441 (+ 4142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.076426 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.076589 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.076818 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.077254 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.077462 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:28:21.078559 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.066139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40083 (local address 127.0.97.1:40925)
0620 02:28:21.067679 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:21.067695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:21.067834 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:21.068247 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:21.068252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:21.068277 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:21.068851 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.068861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.070482 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.070488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.074186 (+ 3698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.074203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.076764 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.076773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.077032 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.077071 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:28:21.077175 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:28:21.077331 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"thread_start_us":1286,"threads_started":1}
I20260620 02:28:21.079727 2024 raft_consensus.cc:504] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:21.080345 2024 leader_election.cc:288] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [CANDIDATE]: Term 1 election: Requested vote from peers 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565), 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083)
I20260620 02:28:21.080969 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0006f199d7f43e0a1fbc99642b0ee00" candidate_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" is_pre_election: true
I20260620 02:28:21.081070 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0006f199d7f43e0a1fbc99642b0ee00" candidate_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3"
I20260620 02:28:21.081409 1809 raft_consensus.cc:3037] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:21.081408 1949 raft_consensus.cc:2454] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7651c8d9f63643e5a0fd825cdac8f6e6 in term 0.
I20260620 02:28:21.081271 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0006f199d7f43e0a1fbc99642b0ee00" candidate_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf9c61a75b14a598a1ef79dc60ca943"
I20260620 02:28:21.081578 1948 raft_consensus.cc:3037] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:21.083995 1948 raft_consensus.cc:2454] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7651c8d9f63643e5a0fd825cdac8f6e6 in term 1.
I20260620 02:28:21.084918 1602 leader_election.cc:302] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7651c8d9f63643e5a0fd825cdac8f6e6, 7bf9c61a75b14a598a1ef79dc60ca943; no voters:
I20260620 02:28:21.085281 2024 raft_consensus.cc:2781] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:21.085541 1809 raft_consensus.cc:2454] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7651c8d9f63643e5a0fd825cdac8f6e6 in term 1.
I20260620 02:28:21.086067 2024 raft_consensus.cc:686] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 LEADER]: Becoming Leader. State: Replica: 7651c8d9f63643e5a0fd825cdac8f6e6, State: Running, Role: LEADER
I20260620 02:28:21.086495 2024 consensus_queue.cc:237] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } }
I20260620 02:28:21.088781 2063 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: TxnStatusTablet state changed. Reason: New leader 7651c8d9f63643e5a0fd825cdac8f6e6. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.089515 2063 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:28:21.091128 1513 catalog_manager.cc:5495] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260620 02:28:21.093025 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:28:21.093350 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:28:21.171960 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.164483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.0.97.1:39703)
0620 02:28:21.164693 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.164701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.165503 (+ 802us) server_negotiation.cc:407] Connection header received
0620 02:28:21.165820 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.165824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.165937 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.166096 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:21.166566 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.167498 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.169406 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.169905 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.170050 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.170609 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.171183 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.171353 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.171473 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.171561 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:28:21.172098 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.164509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39703 (local address 127.0.0.1:51420)
0620 02:28:21.165393 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:21.165410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:21.165591 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:21.166121 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:21.166125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:21.166138 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:21.166396 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.166405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.167637 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.167645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.169253 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.169266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.170091 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.170097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.170341 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.170403 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.171302 (+ 899us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.171326 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:28:21.171446 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:28:21.171632 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720}
I20260620 02:28:21.178391 1809 raft_consensus.cc:1261] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Refusing update from remote peer 7651c8d9f63643e5a0fd825cdac8f6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:21.179507 1948 raft_consensus.cc:1261] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Refusing update from remote peer 7651c8d9f63643e5a0fd825cdac8f6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:21.179396 2063 consensus_queue.cc:1035] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:21.180301 2063 consensus_queue.cc:1035] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:21.190469 2028 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: TxnStatusTablet state changed. Reason: New leader 7651c8d9f63643e5a0fd825cdac8f6e6. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.191045 2028 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:21.195216 2025 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: TxnStatusTablet state changed. Reason: New leader 7651c8d9f63643e5a0fd825cdac8f6e6. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.195576 2025 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:21.201035 2028 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.201458 2028 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7bf9c61a75b14a598a1ef79dc60ca943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:21.206669 2063 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.207253 2063 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:28:21.207713 2063 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.208052 2063 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 7651c8d9f63643e5a0fd825cdac8f6e6: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:28:21.210515 2025 tablet_replica.cc:439] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 } } peers { permanent_uuid: "3c6dee7ac5ae4a1b9434b66817507dc3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39565 } } peers { permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 } } }
I20260620 02:28:21.211037 2025 tablet_replica.cc:442] T b0006f199d7f43e0a1fbc99642b0ee00 P 3c6dee7ac5ae4a1b9434b66817507dc3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:21.213523 2025 consensus_queue.cc:1035] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf9c61a75b14a598a1ef79dc60ca943" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:21.224140 2025 consensus_queue.cc:1035] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7651c8d9f63643e5a0fd825cdac8f6e6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:21.231112 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.219434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.0.97.2:39565)
0620 02:28:21.219697 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.219707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.220582 (+ 875us) server_negotiation.cc:407] Connection header received
0620 02:28:21.220889 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.220895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.221080 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.221228 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:21.222268 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.223836 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.226809 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.227680 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.227797 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.228825 (+ 1028us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.229877 (+ 1052us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.230003 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.230610 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.230705 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:28:21.253984 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.245044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45497 (local address 127.0.97.1:39703)
0620 02:28:21.245361 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.245369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.246540 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 02:28:21.246658 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.246662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.246731 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.246827 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:21.247686 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.248736 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.251479 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.252300 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.252428 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.253161 (+ 733us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.253298 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.253373 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.253567 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.253636 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:28:21.254315 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.245129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39703 (local address 127.0.97.2:45497)
0620 02:28:21.246281 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:21.246318 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:21.246515 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:21.246887 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:21.246892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:21.246906 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:21.247484 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.247508 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.248915 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.248920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.251292 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.251309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.252530 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.252537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.252846 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.252958 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.253415 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.253428 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:28:21.253531 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:28:21.253736 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":629,"threads_started":1}
I20260620 02:28:21.314252 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.298983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39565 (local address 127.0.97.1:43893)
0620 02:28:21.305106 (+ 6123us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:21.305130 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:21.305337 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:21.306044 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:21.306049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:21.306161 (+ 112us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:21.306683 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.306695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.308410 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.308416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.311170 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:21.311187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.312394 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.312401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:21.312619 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.312721 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.313336 (+ 615us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.313349 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:28:21.313447 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:28:21.313668 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5804,"thread_start_us":14038,"threads_started":1}
I20260620 02:28:21.315152 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.298939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43893 (local address 127.0.97.2:39565)
0620 02:28:21.299166 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.299174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.305220 (+ 6046us) server_negotiation.cc:407] Connection header received
0620 02:28:21.305486 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.305491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.305577 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.305654 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:21.306951 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.308220 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.311360 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.312203 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.312337 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.312962 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:21.313170 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:21.313283 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.314754 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.314845 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:28:21.364219 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:21.364569 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:21.369470 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a386e00add740cb9eaa9854f5a5cb69: Sending DeleteTablet for 3 replicas of tablet 1069ea39090d46fba9a8337bd4db9479
I20260620 02:28:21.371546 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1069ea39090d46fba9a8337bd4db9479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43332
I20260620 02:28:21.372151 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 1069ea39090d46fba9a8337bd4db9479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45708
I20260620 02:28:21.376806 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 1069ea39090d46fba9a8337bd4db9479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52636
I20260620 02:28:21.390121 2093 tablet_replica.cc:330] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: stopping tablet replica
I20260620 02:28:21.390501 2093 raft_consensus.cc:2229] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:21.391002 2093 raft_consensus.cc:2258] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:21.392592 2093 ts_tablet_manager.cc:1905] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:21.401716 2091 tablet_replica.cc:330] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: stopping tablet replica
I20260620 02:28:21.402031 2091 raft_consensus.cc:2229] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:21.402367 2091 raft_consensus.cc:2258] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:21.404502 2091 ts_tablet_manager.cc:1905] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:21.406098 2093 ts_tablet_manager.cc:1918] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:21.407193 2090 tablet_replica.cc:330] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: stopping tablet replica
I20260620 02:28:21.407590 2090 raft_consensus.cc:2229] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:21.408635 2090 raft_consensus.cc:2258] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:21.410656 2090 ts_tablet_manager.cc:1905] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:21.406368 2093 log.cc:1196] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/1069ea39090d46fba9a8337bd4db9479
I20260620 02:28:21.419147 2091 ts_tablet_manager.cc:1918] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:21.419386 2091 log.cc:1196] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/1069ea39090d46fba9a8337bd4db9479
I20260620 02:28:21.419998 2093 ts_tablet_manager.cc:1939] T 1069ea39090d46fba9a8337bd4db9479 P 7651c8d9f63643e5a0fd825cdac8f6e6: Deleting consensus metadata
I20260620 02:28:21.420113 2091 ts_tablet_manager.cc:1939] T 1069ea39090d46fba9a8337bd4db9479 P 7bf9c61a75b14a598a1ef79dc60ca943: Deleting consensus metadata
I20260620 02:28:21.421416 1500 catalog_manager.cc:4841] TS 7651c8d9f63643e5a0fd825cdac8f6e6 (127.0.97.1:39703): tablet 1069ea39090d46fba9a8337bd4db9479 (table test-table [id=08b359ec9c284bc8b439b4be248749c8]) successfully deleted
I20260620 02:28:21.432479 1499 catalog_manager.cc:4841] TS 7bf9c61a75b14a598a1ef79dc60ca943 (127.0.97.3:40083): tablet 1069ea39090d46fba9a8337bd4db9479 (table test-table [id=08b359ec9c284bc8b439b4be248749c8]) successfully deleted
I20260620 02:28:21.433833 2090 ts_tablet_manager.cc:1918] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:21.434072 2090 log.cc:1196] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/1069ea39090d46fba9a8337bd4db9479
I20260620 02:28:21.434835 2090 ts_tablet_manager.cc:1939] T 1069ea39090d46fba9a8337bd4db9479 P 3c6dee7ac5ae4a1b9434b66817507dc3: Deleting consensus metadata
I20260620 02:28:21.436422 1499 catalog_manager.cc:4841] TS 3c6dee7ac5ae4a1b9434b66817507dc3 (127.0.97.2:39565): tablet 1069ea39090d46fba9a8337bd4db9479 (table test-table [id=08b359ec9c284bc8b439b4be248749c8]) successfully deleted
Jun 20 02:28:21 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:21.457230 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.384491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.0.97.1:39703)
0620 02:28:21.384718 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.384733 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.427706 (+ 42973us) server_negotiation.cc:407] Connection header received
0620 02:28:21.428055 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.428062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.428165 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.428310 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:21.432038 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.434002 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.437056 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.438063 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.438228 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.445694 (+ 7466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:21.445716 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:21.445737 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:21.445837 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:21.451757 (+ 5920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.453123 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.453134 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.453137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.453407 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.453941 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.453946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.453950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.454523 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:21.454895 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.456387 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.456626 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 02:28:21.458827 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51428
I20260620 02:28:21.509310 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.469794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.0.97.62:40619)
0620 02:28:21.484203 (+ 14409us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.484211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.484264 (+ 53us) server_negotiation.cc:407] Connection header received
0620 02:28:21.484404 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.484410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.484592 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.484757 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:21.486826 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.488474 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.490877 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.491972 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.492111 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.496576 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:21.496586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:21.496591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:21.496638 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:21.503525 (+ 6887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.504923 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.504929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.504933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.505283 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.505928 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.505933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.505937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.506392 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:21.506724 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.508382 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.508718 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14132,"thread_start_us":15124,"threads_started":1}
W20260620 02:28:21.510551 1512 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54356
I20260620 02:28:21.513821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1585
I20260620 02:28:21.537060 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.515142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.0.97.1:39703)
0620 02:28:21.515359 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.515367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.521648 (+ 6281us) server_negotiation.cc:407] Connection header received
0620 02:28:21.521882 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.521887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.521965 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.522086 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:21.523366 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.524149 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.525496 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.526110 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.526259 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.529275 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:21.529281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:21.529284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:21.529323 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:21.533373 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.534199 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.534204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.534206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.534362 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.534829 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.534833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.534835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.535198 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:21.535594 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.536030 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.536329 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:28:21.537808 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51444
W20260620 02:28:21.538471 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:21.538760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1585
I20260620 02:28:21.559999 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1725
Jun 20 02:28:21 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:21.579800 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.560970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.97.2:39565)
0620 02:28:21.561190 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.561196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.561218 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:28:21.561395 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.561401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.561516 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.561630 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:21.562852 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.564309 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.565974 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.566868 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.567013 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.571858 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:21.571880 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:21.571900 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:21.571963 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:21.576332 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.577245 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.577256 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.577260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.577465 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.577946 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.577952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.577955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.578495 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:21.578833 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.579125 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.579400 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 02:28:21.580680 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48764
W20260620 02:28:21.581537 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:21.581722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1725
I20260620 02:28:21.603816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1864
Jun 20 02:28:21 dist-test-slave-r706 krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:21.622622 2099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:21.604927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.0.97.3:40083)
0620 02:28:21.605535 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:21.605540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:21.605562 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:28:21.605627 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:21.605630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:21.605699 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:21.605824 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:21.607543 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.608342 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.610145 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:21.610701 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:21.610852 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:21.615835 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:21.615857 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:21.615874 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:21.615924 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:21.619210 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.620084 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.620092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.620095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.620240 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:21.620670 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:21.620674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:21.620676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:21.621125 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:21.621554 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:21.621970 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 02:28:21.622228 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
W20260620 02:28:21.624118 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51806
W20260620 02:28:21.625063 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:21.625295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1864
I20260620 02:28:21.647188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 1482
I20260620 02:28:21.648460 1554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:21.765103 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 1482
2026-06-20T02:28:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5233 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:21 dist-test-slave-r706 krb5kdc[2103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:21 dist-test-slave-r706 krb5kdc[2103](info): set up 2 sockets
Jun 20 02:28:21 dist-test-slave-r706 krb5kdc[2103](info): commencing operation
krb5kdc: starting...
W20260620 02:28:22.857308 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:22 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:23.065408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38643
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:23.401821 2116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:23.402225 2116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:23.402411 2116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:23.409808 2116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:23.409981 2116 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:28:23.410058 2116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:23.410152 2116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:23.410234 2116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:23.410326 2116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:23.420527 2116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38643
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:23.421944 2116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:23.423381 2116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:23.435808 2122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:23.435883 2124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:23.435900 2121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:23.436625 2116 server_base.cc:1034] running on GCE node
I20260620 02:28:23.437508 2116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:23.439059 2116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:23.440291 2116 hybrid_clock.cc:648] HybridClock initialized: now 1781922503440211 us; error 59 us; skew 500 ppm
I20260620 02:28:23.440671 2116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:23 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:23.444039 2116 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:23.445653 2116 webserver.cc:460] Webserver started at http://127.0.97.62:46137/ using document root <none> and password file <none>
I20260620 02:28:23.446421 2116 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:23.446547 2116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:23.446910 2116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:23.450126 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "e883322978fb4e3e82c3d30791e60b71"
format_stamp: "Formatted at 2026-06-20 02:28:23 on dist-test-slave-r706"
server_key: "87947a71c5ad1f9a622d0ecacabc1a15"
server_key_iv: "7afbb396ed9a6782d18b5527486a1bbf"
server_key_version: "encryptionkey@0"
I20260620 02:28:23.450968 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "e883322978fb4e3e82c3d30791e60b71"
format_stamp: "Formatted at 2026-06-20 02:28:23 on dist-test-slave-r706"
server_key: "87947a71c5ad1f9a622d0ecacabc1a15"
server_key_iv: "7afbb396ed9a6782d18b5527486a1bbf"
server_key_version: "encryptionkey@0"
I20260620 02:28:23.456681 2116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:28:23.460592 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:23.462159 2116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:28:23.462357 2116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "e883322978fb4e3e82c3d30791e60b71"
format_stamp: "Formatted at 2026-06-20 02:28:23 on dist-test-slave-r706"
server_key: "87947a71c5ad1f9a622d0ecacabc1a15"
server_key_iv: "7afbb396ed9a6782d18b5527486a1bbf"
server_key_version: "encryptionkey@0"
I20260620 02:28:23.462602 2116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:23.489840 2116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:23.494652 2116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:23.494966 2116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:23.516939 2116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38643
I20260620 02:28:23.516957 2192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38643 every 8 connection(s)
I20260620 02:28:23.518457 2116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:23.522526 2193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:23.525992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 2116
I20260620 02:28:23.526228 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:23.527086 388 external_mini_cluster.cc:1428] Setting key adbe505bef8735b0480724e0e096303f
I20260620 02:28:23.535776 2193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Bootstrap starting.
I20260620 02:28:23.539108 2193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:23.540170 2193 log.cc:824] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:28:23 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:23.543548 2193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: No bootstrap required, opened a new log
I20260620 02:28:23.551914 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:23.529520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.0.97.62:38643)
0620 02:28:23.530312 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:23.530324 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:23.530368 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:28:23.532144 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:23.532168 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:23.532611 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:23.533076 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:23.534620 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:23.535964 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:23.537358 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:23.538080 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:23.538292 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:23.542521 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:23.542533 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:23.542555 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:23.542607 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:23.546763 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:23.547607 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:23.547617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:23.547627 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:23.547877 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:23.548288 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:23.548295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:23.548303 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:23.549106 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:23.549428 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:23.549880 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 02:28:23.550391 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":343,"threads_started":1}
I20260620 02:28:23.554527 2193 raft_consensus.cc:348] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e883322978fb4e3e82c3d30791e60b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38643 } }
I20260620 02:28:23.554944 2193 raft_consensus.cc:374] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:23.555079 2193 raft_consensus.cc:729] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e883322978fb4e3e82c3d30791e60b71, State: Initialized, Role: FOLLOWER
I20260620 02:28:23.555868 2193 consensus_queue.cc:260] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e883322978fb4e3e82c3d30791e60b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38643 } }
I20260620 02:28:23.556135 2193 raft_consensus.cc:388] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:23.556293 2193 raft_consensus.cc:482] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:23.556469 2193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:23.559401 2193 raft_consensus.cc:504] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e883322978fb4e3e82c3d30791e60b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38643 } }
I20260620 02:28:23.560104 2193 leader_election.cc:302] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e883322978fb4e3e82c3d30791e60b71; no voters:
I20260620 02:28:23.563206 2193 leader_election.cc:288] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:23.563464 2198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:23.564332 2198 raft_consensus.cc:686] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [term 1 LEADER]: Becoming Leader. State: Replica: e883322978fb4e3e82c3d30791e60b71, State: Running, Role: LEADER
I20260620 02:28:23.564939 2198 consensus_queue.cc:237] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e883322978fb4e3e82c3d30791e60b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38643 } }
I20260620 02:28:23.566782 2193 sys_catalog.cc:564] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:23.568425 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e883322978fb4e3e82c3d30791e60b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e883322978fb4e3e82c3d30791e60b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38643 } } }
I20260620 02:28:23.568511 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e883322978fb4e3e82c3d30791e60b71. Latest consensus state: current_term: 1 leader_uuid: "e883322978fb4e3e82c3d30791e60b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e883322978fb4e3e82c3d30791e60b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38643 } } }
I20260620 02:28:23.568845 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:23.568895 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:23.571357 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:23.577765 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:23.587126 2203 catalog_manager.cc:1347] Generated new cluster ID: f84fb6619d5a4638b43f6d1686b87876
I20260620 02:28:23.587281 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:23.617684 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:23.619146 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:23.626122 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Generated new TSK 0
I20260620 02:28:23.627070 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260620 02:28:23.893249 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.251s user 0.000s sys 0.012s
I20260620 02:28:23.894328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38643
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:24.240887 2219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:24.241343 2219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:24.241552 2219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:24.251818 2219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:24.251993 2219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:24.252175 2219 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:28:24.252271 2219 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:28:24.252406 2219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:24.264972 2219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:24.266467 2219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:24.267948 2219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:24.282979 2224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:24.283084 2225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:24.283103 2227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:24.283499 2219 server_base.cc:1034] running on GCE node
I20260620 02:28:24.284276 2219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:24.285334 2219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:24.286590 2219 hybrid_clock.cc:648] HybridClock initialized: now 1781922504286559 us; error 30 us; skew 500 ppm
I20260620 02:28:24.286964 2219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:24 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:24.290393 2219 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:24.292006 2219 webserver.cc:460] Webserver started at http://127.0.97.1:45823/ using document root <none> and password file <none>
I20260620 02:28:24.292698 2219 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:24.292805 2219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:24.293151 2219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:24.296105 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "a2469b27d35849a980836e9baa0abac0"
format_stamp: "Formatted at 2026-06-20 02:28:24 on dist-test-slave-r706"
server_key: "33d6601aaf7127eb143f547ac0670c49"
server_key_iv: "2a5aeff432b426c7ff3eb8a8ad6a419f"
server_key_version: "encryptionkey@0"
I20260620 02:28:24.296873 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a2469b27d35849a980836e9baa0abac0"
format_stamp: "Formatted at 2026-06-20 02:28:24 on dist-test-slave-r706"
server_key: "33d6601aaf7127eb143f547ac0670c49"
server_key_iv: "2a5aeff432b426c7ff3eb8a8ad6a419f"
server_key_version: "encryptionkey@0"
I20260620 02:28:24.302332 2219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:28:24.306033 2234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:24.307626 2219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:24.307803 2219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "a2469b27d35849a980836e9baa0abac0"
format_stamp: "Formatted at 2026-06-20 02:28:24 on dist-test-slave-r706"
server_key: "33d6601aaf7127eb143f547ac0670c49"
server_key_iv: "2a5aeff432b426c7ff3eb8a8ad6a419f"
server_key_version: "encryptionkey@0"
I20260620 02:28:24.308009 2219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:24.320204 2219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:24.325352 2219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:24.325660 2219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:24.328613 2219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:24.328792 2219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:24.328943 2219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:24.329071 2219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:28:24 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:24.353472 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.329039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40589 (local address 127.0.97.62:38643)
0620 02:28:24.329672 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:24.329680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:24.330977 (+ 1297us) server_negotiation.cc:407] Connection header received
0620 02:28:24.334257 (+ 3280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:24.334266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:24.334380 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:24.334522 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:24.337776 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.338758 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.340466 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.340883 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.341054 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.346151 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:24.346160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:24.346170 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:24.346231 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:24.350000 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.350985 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.350991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.350995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.351131 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.351600 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.351605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.351609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.351891 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:24.352053 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:24.352811 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 02:28:24.353082 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260620 02:28:24.354857 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.329785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.97.1:40589)
0620 02:28:24.330743 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:24.330812 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:24.333795 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:24.334889 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:24.334903 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:24.336029 (+ 1126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:24.337476 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.337523 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.338939 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.338950 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.340300 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.340314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.341180 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.341187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.341509 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.342416 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:24.342439 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:24.345744 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:24.350230 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.350247 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.350281 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.350834 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.351249 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.351257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.351263 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.351472 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.352030 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:24.352038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:24.352418 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 02:28:24.352695 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 02:28:24.353328 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":359,"threads_started":1}
I20260620 02:28:24.379253 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.362268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58967 (local address 127.0.97.62:38643)
0620 02:28:24.362516 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:24.362525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:24.363350 (+ 825us) server_negotiation.cc:407] Connection header received
0620 02:28:24.363592 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:24.363598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:24.363668 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:24.363764 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:24.365723 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.366697 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.368247 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.368682 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.368828 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.371955 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:24.371962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:24.371968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:24.372018 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:24.375600 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.376931 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.376937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.376939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.377176 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.377651 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.377658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.377663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.377995 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:24.378168 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:24.378635 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 02:28:24.378884 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:28:24.379364 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.362294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.97.1:58967)
0620 02:28:24.363194 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:24.363225 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:24.363446 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:24.363847 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:24.363856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:24.364741 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:24.365447 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.365468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.366837 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.366843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.368095 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.368108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.368875 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.368881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.369097 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.369863 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:24.369907 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:24.371695 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:24.375803 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.375813 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.375822 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.376711 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.377323 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.377328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.377332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.377497 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.378126 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:24.378131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:24.378356 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 02:28:24.378512 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:28:24.378888 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":440,"threads_started":1}
I20260620 02:28:24.391690 2219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39829
I20260620 02:28:24.391896 2354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39829 every 8 connection(s)
I20260620 02:28:24.393214 2219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:24.395865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 2219
I20260620 02:28:24.395853 2355 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38643
I20260620 02:28:24.396073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:24.396123 2355 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:24.396942 388 external_mini_cluster.cc:1428] Setting key 19fc4a30855b0dc13e157e50ea4d2663
I20260620 02:28:24.397030 2355 heartbeater.cc:511] Master 127.0.97.62:38643 requested a full tablet report, sending...
I20260620 02:28:24.399528 2147 ts_manager.cc:194] Registered new tserver with Master: a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829)
I20260620 02:28:24.401907 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40589
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:24.445199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38643
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:24.798089 2359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:24.798465 2359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:24.798659 2359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:24.806936 2359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:24.807109 2359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:24.807264 2359 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:28:24.807354 2359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:28:24.807490 2359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:24.818280 2359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:24.819715 2359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:24.821242 2359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:24.838920 2364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:24.838858 2367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:24.838907 2365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:24.840175 2359 server_base.cc:1034] running on GCE node
I20260620 02:28:24.840904 2359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:24.842029 2359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:24.843279 2359 hybrid_clock.cc:648] HybridClock initialized: now 1781922504843233 us; error 48 us; skew 500 ppm
I20260620 02:28:24.843636 2359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:24 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:24.847057 2359 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:24.848671 2359 webserver.cc:460] Webserver started at http://127.0.97.2:34227/ using document root <none> and password file <none>
I20260620 02:28:24.849555 2359 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:24.849712 2359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:24.850044 2359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:24.853314 2359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "439aab12911b4b71b04dc8472d20d731"
format_stamp: "Formatted at 2026-06-20 02:28:24 on dist-test-slave-r706"
server_key: "4c260bea779bb5fefb0b16bec3ae6202"
server_key_iv: "21ecb71de746e2e6948d6970004ee02f"
server_key_version: "encryptionkey@0"
I20260620 02:28:24.854226 2359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "439aab12911b4b71b04dc8472d20d731"
format_stamp: "Formatted at 2026-06-20 02:28:24 on dist-test-slave-r706"
server_key: "4c260bea779bb5fefb0b16bec3ae6202"
server_key_iv: "21ecb71de746e2e6948d6970004ee02f"
server_key_version: "encryptionkey@0"
I20260620 02:28:24.860345 2359 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:28:24.864847 2374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:24.866654 2359 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:28:24.866905 2359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "439aab12911b4b71b04dc8472d20d731"
format_stamp: "Formatted at 2026-06-20 02:28:24 on dist-test-slave-r706"
server_key: "4c260bea779bb5fefb0b16bec3ae6202"
server_key_iv: "21ecb71de746e2e6948d6970004ee02f"
server_key_version: "encryptionkey@0"
I20260620 02:28:24.867173 2359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:24.888754 2359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:24.893814 2359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:24.894124 2359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:24.897382 2359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:24.897578 2359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:24.897722 2359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:24.897816 2359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:28:24 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:24.921741 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.897748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59735 (local address 127.0.97.62:38643)
0620 02:28:24.898271 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:24.898277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:24.899609 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 02:28:24.902026 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:24.902035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:24.902139 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:24.902296 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:24.904443 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.905780 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.908582 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.909267 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.909422 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.913935 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:24.913941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:24.913945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:24.913983 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:24.917909 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.919075 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.919084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.919088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.919235 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.919718 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.919723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.919727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.920042 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:24.920200 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:24.921049 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 02:28:24.921308 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
I20260620 02:28:24.923161 2384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.898434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.97.2:59735)
0620 02:28:24.899406 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:24.899455 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:24.901653 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:24.902500 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:24.902509 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:24.903186 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:24.904189 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.904228 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.905996 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.906005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.908389 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.908414 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.909541 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.909549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.909835 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.910733 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:24.910761 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:24.913669 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:24.918091 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.918102 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.918122 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.918895 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.919368 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.919373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.919377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.919594 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.920189 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:24.920198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:24.920622 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 02:28:24.920924 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 02:28:24.921587 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":398,"threads_started":1}
I20260620 02:28:24.946130 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.929622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59537 (local address 127.0.97.62:38643)
0620 02:28:24.929805 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:24.929809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:24.930563 (+ 754us) server_negotiation.cc:407] Connection header received
0620 02:28:24.930805 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:24.930811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:24.930891 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:24.931002 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:24.931964 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.932714 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.934114 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.934683 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.934846 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.937889 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:24.937894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:24.937898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:24.937960 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:24.941615 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.942963 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.942971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.942976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.943158 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:24.943878 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:24.943884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:24.943889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:24.944437 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:24.944552 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:24.945488 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 02:28:24.945769 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:28:24.946283 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:24.929697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.97.2:59537)
0620 02:28:24.930424 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:24.930442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:24.930665 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:24.931028 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:24.931032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:24.931513 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:24.931807 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.931816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.932855 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.932863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.933956 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:24.933969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:24.935142 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:24.935148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:24.935334 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:24.936353 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:24.936372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:24.937629 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:24.941811 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.941817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.941821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.942721 (+ 900us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.943451 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:24.943465 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:24.943471 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:24.943682 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:24.944694 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:24.944702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:24.945194 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 02:28:24.945381 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 02:28:24.945798 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":453,"threads_started":1}
I20260620 02:28:24.959939 2359 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41561
I20260620 02:28:24.959995 2494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41561 every 8 connection(s)
I20260620 02:28:24.961675 2359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:24.964819 2495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38643
I20260620 02:28:24.965145 2495 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:24.965862 2495 heartbeater.cc:511] Master 127.0.97.62:38643 requested a full tablet report, sending...
I20260620 02:28:24.967226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 2359
I20260620 02:28:24.967406 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:24.967702 2147 ts_manager.cc:194] Registered new tserver with Master: 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561)
I20260620 02:28:24.968015 388 external_mini_cluster.cc:1428] Setting key 660c21c05db19fd4d1213c94e9844828
I20260620 02:28:24.968669 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59735
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:25.012915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38643
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:25.360544 2499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:25.360899 2499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:25.361146 2499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:25.368789 2499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:25.368955 2499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:25.369135 2499 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:28:25.369247 2499 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:28:25.369386 2499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:25.379766 2499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:25.381269 2499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:25.382655 2499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:25.396508 2504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:25.396508 2505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:25.396719 2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:25.398008 2499 server_base.cc:1034] running on GCE node
I20260620 02:28:25.398805 2499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:25.399923 2499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:25.401188 2499 hybrid_clock.cc:648] HybridClock initialized: now 1781922505401139 us; error 72 us; skew 500 ppm
I20260620 02:28:25.401571 2499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:25 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:25.405176 2499 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:25.405192 2355 heartbeater.cc:503] Master 127.0.97.62:38643 was elected leader, sending a full tablet report...
I20260620 02:28:25.407096 2499 webserver.cc:460] Webserver started at http://127.0.97.3:33179/ using document root <none> and password file <none>
I20260620 02:28:25.407833 2499 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:25.407994 2499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:25.408329 2499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:25.411710 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "e402dd8bc41e44efa166b3bd488602ab"
format_stamp: "Formatted at 2026-06-20 02:28:25 on dist-test-slave-r706"
server_key: "e1e0a5ba07d66e6f5679c7bead864364"
server_key_iv: "d163af412d35a70cc61e63f9b0d8267b"
server_key_version: "encryptionkey@0"
I20260620 02:28:25.412674 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e402dd8bc41e44efa166b3bd488602ab"
format_stamp: "Formatted at 2026-06-20 02:28:25 on dist-test-slave-r706"
server_key: "e1e0a5ba07d66e6f5679c7bead864364"
server_key_iv: "d163af412d35a70cc61e63f9b0d8267b"
server_key_version: "encryptionkey@0"
I20260620 02:28:25.418557 2499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 02:28:25.422372 2514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:25.424019 2499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:25.424254 2499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "e402dd8bc41e44efa166b3bd488602ab"
format_stamp: "Formatted at 2026-06-20 02:28:25 on dist-test-slave-r706"
server_key: "e1e0a5ba07d66e6f5679c7bead864364"
server_key_iv: "d163af412d35a70cc61e63f9b0d8267b"
server_key_version: "encryptionkey@0"
I20260620 02:28:25.424525 2499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:25.455708 2499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:25.460947 2499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:25.461339 2499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:25.464246 2499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:25.464392 2499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:25.464521 2499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:25.464609 2499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:28:25 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:25.490381 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.465062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40957 (local address 127.0.97.62:38643)
0620 02:28:25.465475 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.465479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.467644 (+ 2165us) server_negotiation.cc:407] Connection header received
0620 02:28:25.470352 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.470359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.470459 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.470575 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:25.473419 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.474207 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.475899 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.476312 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.476461 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.481643 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:25.481652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:25.481659 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:25.481702 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:25.485990 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.487064 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.487075 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.487081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.487281 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.487792 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.487800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.487805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.488250 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:25.488432 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.489852 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.490076 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260620 02:28:25.492408 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.466266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.97.3:40957)
0620 02:28:25.467385 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.467446 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.469981 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.470847 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.470856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.472015 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:25.473177 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.473212 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.474367 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.474373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.475733 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.475746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.476550 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.476566 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.476791 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.477703 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:25.477749 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:25.481253 (+ 3504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:25.486207 (+ 4954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:25.486222 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:25.486248 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:25.486876 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:25.487425 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:25.487431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:25.487434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:25.487653 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:25.488528 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:25.488542 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:25.489252 (+ 710us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.489679 (+ 427us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.490477 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":404,"threads_started":1}
I20260620 02:28:25.515486 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.499627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39183 (local address 127.0.97.62:38643)
0620 02:28:25.499857 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.499863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.499885 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:28:25.500090 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.500099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.500212 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.500331 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:25.501505 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.502551 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.504475 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.505490 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.505663 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.507496 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:25.507506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:25.507514 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:25.507572 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:25.511907 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.512904 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.512910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.512913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.513114 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.513650 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.513660 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.513666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.514220 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:25.514401 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.514859 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.515118 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:28:25.515661 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.498908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.97.3:39183)
0620 02:28:25.499716 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.499745 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.499921 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.500388 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.500393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.500872 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:25.501269 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.501281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.502768 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.502772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.504320 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.504332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.505692 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.505697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.505813 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.506470 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:25.506489 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:25.507266 (+ 777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:25.512072 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:25.512077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:25.512080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:25.512675 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:25.513240 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:25.513245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:25.513249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:25.513472 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:25.514373 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:25.514376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:25.514576 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.514760 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.515199 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":453,"threads_started":1}
I20260620 02:28:25.525789 2499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43265
I20260620 02:28:25.525875 2634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43265 every 8 connection(s)
I20260620 02:28:25.527158 2499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:25.529500 2635 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38643
I20260620 02:28:25.529754 2635 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:25.530418 2635 heartbeater.cc:511] Master 127.0.97.62:38643 requested a full tablet report, sending...
I20260620 02:28:25.531934 2148 ts_manager.cc:194] Registered new tserver with Master: e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265)
I20260620 02:28:25.532913 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40957
I20260620 02:28:25.534723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 2499
I20260620 02:28:25.534893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:25.535359 388 external_mini_cluster.cc:1428] Setting key cbca8f902dfc44457c53ed9487ac694e
I20260620 02:28:25.538784 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:28:25 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:28:25 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:25.588894 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.573889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.0.97.62:38643)
0620 02:28:25.574129 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.574136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.574334 (+ 198us) server_negotiation.cc:407] Connection header received
0620 02:28:25.574560 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.574572 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.574640 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.574751 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:25.576183 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.576941 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.578251 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.578661 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.578799 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.582955 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:25.582960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:25.582963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:25.583001 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:25.585754 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.586438 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.586444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.586450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.587001 (+ 551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.587404 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.587410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.587413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.587715 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:25.587988 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.588294 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.588545 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:28:25.593401 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:25.600508 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:25.630828 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.613348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.0.97.2:41561)
0620 02:28:25.617589 (+ 4241us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.617601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.618702 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 02:28:25.618811 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.618818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.619078 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.619267 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:25.620617 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.622573 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.626594 (+ 4021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.628661 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.628804 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.629154 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.630106 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.630377 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4028,"thread_start_us":3880,"threads_started":1}
I20260620 02:28:25.631649 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.613961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.0.97.3:43265)
0620 02:28:25.614866 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.614880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.618402 (+ 3522us) server_negotiation.cc:407] Connection header received
0620 02:28:25.618746 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.618757 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.619097 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.619301 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:25.620802 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.623097 (+ 2295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.626904 (+ 3807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.629571 (+ 2667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.629717 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.630011 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.630893 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.631154 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":398,"threads_started":1}
I20260620 02:28:25.631987 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.613085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41561 (local address 127.0.0.1:36996)
0620 02:28:25.618215 (+ 5130us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.618278 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.618472 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.619609 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.619621 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.619665 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:25.620337 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.620354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.622791 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.622799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.626331 (+ 3532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.626356 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.629420 (+ 3064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.629430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.629765 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.629817 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.631156 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.631467 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4863,"mutex_wait_us":37,"thread_start_us":4864,"threads_started":1}
I20260620 02:28:25.632851 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.613358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43265 (local address 127.0.0.1:48822)
0620 02:28:25.618247 (+ 4889us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.618276 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.618535 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.619786 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.619794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.619837 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:25.620515 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.620531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.623353 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.623361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.626642 (+ 3281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.626666 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.630282 (+ 3616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.630288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.630584 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.630639 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.632036 (+ 1397us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.632261 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4738,"mutex_wait_us":87,"spinlock_wait_cycles":5376,"thread_start_us":4300,"threads_started":1}
I20260620 02:28:25.636793 2565 tablet_service.cc:1467] Processing CreateTablet for tablet 1273c9f87f204936aab2ade9c8646268 (DEFAULT_TABLE table=test-table [id=4518dfda786d44419dd5db8c6092db8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:25.638527 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1273c9f87f204936aab2ade9c8646268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:25.641858 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 1273c9f87f204936aab2ade9c8646268 (DEFAULT_TABLE table=test-table [id=4518dfda786d44419dd5db8c6092db8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:25.643342 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1273c9f87f204936aab2ade9c8646268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:25.644943 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.618679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39829 (local address 127.0.0.1:36014)
0620 02:28:25.632307 (+ 13628us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.632333 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.632490 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.637858 (+ 5368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.637864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.637897 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:25.638114 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.638129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.640046 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.640052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.641935 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.641950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.644079 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.644087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.644295 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.644335 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.644439 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.644617 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13485,"thread_start_us":19606,"threads_started":1}
I20260620 02:28:25.645817 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.619178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.97.1:39829)
0620 02:28:25.634217 (+ 15039us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.634226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.634257 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:28:25.634359 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.637151 (+ 2792us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.637574 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.637850 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:25.638321 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.639822 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.642378 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.643873 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.644030 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.644235 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.645225 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.645423 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14801,"thread_start_us":17890,"threads_started":1}
I20260620 02:28:25.649816 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 1273c9f87f204936aab2ade9c8646268 (DEFAULT_TABLE table=test-table [id=4518dfda786d44419dd5db8c6092db8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:25.651320 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1273c9f87f204936aab2ade9c8646268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:25.653506 2654 tablet_bootstrap.cc:492] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Bootstrap starting.
I20260620 02:28:25.656363 2654 tablet_bootstrap.cc:654] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:25.657755 2654 log.cc:824] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:25.659756 2655 tablet_bootstrap.cc:492] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Bootstrap starting.
I20260620 02:28:25.662701 2654 tablet_bootstrap.cc:492] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: No bootstrap required, opened a new log
I20260620 02:28:25.662994 2654 ts_tablet_manager.cc:1397] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 02:28:25.665432 2655 tablet_bootstrap.cc:654] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:25.666370 2657 tablet_bootstrap.cc:492] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Bootstrap starting.
I20260620 02:28:25.666875 2655 log.cc:824] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:25.670360 2657 tablet_bootstrap.cc:654] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:25.670399 2655 tablet_bootstrap.cc:492] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: No bootstrap required, opened a new log
I20260620 02:28:25.670679 2655 ts_tablet_manager.cc:1397] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 02:28:25.671706 2657 log.cc:824] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:25.674273 2654 raft_consensus.cc:348] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.674765 2654 raft_consensus.cc:374] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:25.674863 2654 raft_consensus.cc:729] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Initialized, Role: FOLLOWER
I20260620 02:28:25.675544 2654 consensus_queue.cc:260] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.676851 2635 heartbeater.cc:503] Master 127.0.97.62:38643 was elected leader, sending a full tablet report...
I20260620 02:28:25.677575 2654 ts_tablet_manager.cc:1428] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
I20260620 02:28:25.681185 2657 tablet_bootstrap.cc:492] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: No bootstrap required, opened a new log
I20260620 02:28:25.681526 2657 ts_tablet_manager.cc:1397] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260620 02:28:25.682243 2655 raft_consensus.cc:348] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.682736 2655 raft_consensus.cc:374] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:25.682849 2655 raft_consensus.cc:729] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Initialized, Role: FOLLOWER
I20260620 02:28:25.683506 2655 consensus_queue.cc:260] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.684634 2495 heartbeater.cc:503] Master 127.0.97.62:38643 was elected leader, sending a full tablet report...
I20260620 02:28:25.685369 2655 ts_tablet_manager.cc:1428] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260620 02:28:25.691004 2657 raft_consensus.cc:348] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.691366 2657 raft_consensus.cc:374] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:25.691440 2657 raft_consensus.cc:729] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2469b27d35849a980836e9baa0abac0, State: Initialized, Role: FOLLOWER
I20260620 02:28:25.692023 2657 consensus_queue.cc:260] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.693962 2657 ts_tablet_manager.cc:1428] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260620 02:28:25.714879 2496 tablet.cc:2369] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:25.751092 2661 raft_consensus.cc:482] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:25.751406 2661 raft_consensus.cc:504] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.753139 2661 leader_election.cc:288] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265), a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829)
I20260620 02:28:25.760588 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.753548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43859 (local address 127.0.97.1:39829)
0620 02:28:25.753769 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.753777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.754707 (+ 930us) server_negotiation.cc:407] Connection header received
0620 02:28:25.754909 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.754914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.755004 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.755109 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:25.755788 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.756492 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.758713 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.759525 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.759693 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.759816 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.760173 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.760310 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:28:25.760743 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.753617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59579 (local address 127.0.97.3:43265)
0620 02:28:25.753854 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.753863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.754641 (+ 778us) server_negotiation.cc:407] Connection header received
0620 02:28:25.754886 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.754892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.754951 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.755073 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:25.755769 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.756492 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.758911 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.759712 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.759894 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.760023 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.760349 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.760478 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:28:25.760799 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.753557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39829 (local address 127.0.97.2:43859)
0620 02:28:25.754619 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.754634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.754748 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.755121 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.755126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.755148 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:25.755627 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.755636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.756632 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.756639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.758529 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.758545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.759775 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.759781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.759969 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.760025 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.760145 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.760306 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":53,"thread_start_us":647,"threads_started":1}
I20260620 02:28:25.760910 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.753510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43265 (local address 127.0.97.2:59579)
0620 02:28:25.754496 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.754523 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.754720 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.755099 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.755103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.755133 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:25.755625 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.755634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.756630 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.756637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.758747 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.758761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.759974 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.759981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.760182 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.760206 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.760310 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.760460 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":147,"thread_start_us":545,"threads_started":1}
I20260620 02:28:25.762265 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1273c9f87f204936aab2ade9c8646268" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab" is_pre_election: true
I20260620 02:28:25.762265 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1273c9f87f204936aab2ade9c8646268" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0" is_pre_election: true
I20260620 02:28:25.762882 2310 raft_consensus.cc:2454] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 0.
I20260620 02:28:25.762881 2589 raft_consensus.cc:2454] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 0.
I20260620 02:28:25.763582 2375 leader_election.cc:302] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, a2469b27d35849a980836e9baa0abac0; no voters:
I20260620 02:28:25.763947 2661 raft_consensus.cc:2781] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:25.764139 2661 raft_consensus.cc:482] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:25.764253 2661 raft_consensus.cc:3037] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:25.766695 2661 raft_consensus.cc:504] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.767383 2661 leader_election.cc:288] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 election: Requested vote from peers e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265), a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829)
I20260620 02:28:25.767863 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1273c9f87f204936aab2ade9c8646268" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab"
I20260620 02:28:25.768119 2589 raft_consensus.cc:3037] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:25.770488 2589 raft_consensus.cc:2454] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 1.
I20260620 02:28:25.770918 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1273c9f87f204936aab2ade9c8646268" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0"
I20260620 02:28:25.771071 2376 leader_election.cc:302] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:25.771184 2310 raft_consensus.cc:3037] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:25.771453 2661 raft_consensus.cc:2781] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:25.772095 2661 raft_consensus.cc:686] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 1 LEADER]: Becoming Leader. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Running, Role: LEADER
I20260620 02:28:25.773042 2661 consensus_queue.cc:237] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } }
I20260620 02:28:25.774395 2310 raft_consensus.cc:2454] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 1.
I20260620 02:28:25.777987 2148 catalog_manager.cc:5495] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 reported cstate change: term changed from 0 to 1, leader changed from <none> to 439aab12911b4b71b04dc8472d20d731 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } health_report { overall_health: UNKNOWN } } }
W20260620 02:28:25.778777 2636 tablet.cc:2369] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:28:25 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:25.833993 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.819839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.0.97.62:38643)
0620 02:28:25.820037 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:25.820042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:25.820519 (+ 477us) server_negotiation.cc:407] Connection header received
0620 02:28:25.820742 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:25.820747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:25.820811 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:25.820917 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:25.822128 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.822878 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.824134 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.824577 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.824719 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.828352 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:25.828357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:25.828360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:25.828389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:25.831110 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.831853 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.831861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.831867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.831997 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:25.832385 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:25.832392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:25.832395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:25.832666 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:25.832818 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:25.833310 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 02:28:25.833562 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":67}
I20260620 02:28:25.834107 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:25.819827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38643 (local address 127.0.0.1:40790)
0620 02:28:25.820407 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:25.820423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:25.820606 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:25.820956 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:25.820961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:25.821704 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:25.821982 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.821991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.823031 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.823038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.823994 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:25.824005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:25.824764 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:25.824770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:25.824921 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:25.825673 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:25.825696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:25.828160 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:25.831281 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:25.831296 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:25.831304 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:25.831731 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:25.832107 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:25.832114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:25.832117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:25.832276 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:25.832792 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:25.832800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:25.832971 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 02:28:25.833223 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 02:28:25.833679 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":173,"thread_start_us":334,"threads_started":1}
I20260620 02:28:25.839151 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:28:25.840078 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:25.846765 2425 tablet_service.cc:1467] Processing CreateTablet for tablet b02da6bdfe4648e99118631378a37632 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29af725f248a4dfcac751d99a9392bf7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:28:25.846765 2285 tablet_service.cc:1467] Processing CreateTablet for tablet b02da6bdfe4648e99118631378a37632 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29af725f248a4dfcac751d99a9392bf7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:28:25.847031 2565 tablet_service.cc:1467] Processing CreateTablet for tablet b02da6bdfe4648e99118631378a37632 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29af725f248a4dfcac751d99a9392bf7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:28:25.847782 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02da6bdfe4648e99118631378a37632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:25.847782 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02da6bdfe4648e99118631378a37632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:25.847859 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02da6bdfe4648e99118631378a37632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:25.853951 2654 tablet_bootstrap.cc:492] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: Bootstrap starting.
I20260620 02:28:25.855083 2657 tablet_bootstrap.cc:492] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: Bootstrap starting.
I20260620 02:28:25.855535 2654 tablet_bootstrap.cc:654] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:25.856863 2657 tablet_bootstrap.cc:654] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:25.857121 2655 tablet_bootstrap.cc:492] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: Bootstrap starting.
I20260620 02:28:25.858458 2654 tablet_bootstrap.cc:492] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: No bootstrap required, opened a new log
I20260620 02:28:25.858616 2654 ts_tablet_manager.cc:1397] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:28:25.859032 2655 tablet_bootstrap.cc:654] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:25.859441 2654 raft_consensus.cc:348] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:25.859673 2654 raft_consensus.cc:374] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:25.859751 2657 tablet_bootstrap.cc:492] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: No bootstrap required, opened a new log
I20260620 02:28:25.859772 2654 raft_consensus.cc:729] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Initialized, Role: FOLLOWER
I20260620 02:28:25.859922 2657 ts_tablet_manager.cc:1397] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:28:25.860075 2654 consensus_queue.cc:260] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:25.860549 2660 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:25.860905 2660 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:25.860946 2657 raft_consensus.cc:348] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:25.861301 2657 raft_consensus.cc:374] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:25.861423 2657 raft_consensus.cc:729] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2469b27d35849a980836e9baa0abac0, State: Initialized, Role: FOLLOWER
I20260620 02:28:25.861675 2654 ts_tablet_manager.cc:1428] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 02:28:25.861788 2657 consensus_queue.cc:260] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:25.862591 2657 ts_tablet_manager.cc:1428] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:25.862388 2662 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:25.862811 2662 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:25.873502 2655 tablet_bootstrap.cc:492] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: No bootstrap required, opened a new log
I20260620 02:28:25.873651 2655 ts_tablet_manager.cc:1397] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260620 02:28:25.874287 2655 raft_consensus.cc:348] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:25.874541 2655 raft_consensus.cc:374] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:25.874653 2655 raft_consensus.cc:729] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Initialized, Role: FOLLOWER
I20260620 02:28:25.874902 2655 consensus_queue.cc:260] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:25.875569 2655 ts_tablet_manager.cc:1428] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:28:25.875357 2661 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:25.875727 2661 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 02:28:25.895977 2356 tablet.cc:2369] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:26.011549 2661 raft_consensus.cc:482] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:26.011763 2661 raft_consensus.cc:504] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.012466 2661 leader_election.cc:288] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265)
I20260620 02:28:26.013141 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02da6bdfe4648e99118631378a37632" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0" is_pre_election: true
I20260620 02:28:26.013156 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02da6bdfe4648e99118631378a37632" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab" is_pre_election: true
I20260620 02:28:26.013411 2589 raft_consensus.cc:2454] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 0.
I20260620 02:28:26.013413 2310 raft_consensus.cc:2454] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 0.
I20260620 02:28:26.013899 2376 leader_election.cc:302] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:26.014251 2661 raft_consensus.cc:2781] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:26.014338 2661 raft_consensus.cc:482] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:26.014477 2661 raft_consensus.cc:3037] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:26.016656 2661 raft_consensus.cc:504] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.017431 2661 leader_election.cc:288] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 election: Requested vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265)
I20260620 02:28:26.017915 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02da6bdfe4648e99118631378a37632" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0"
I20260620 02:28:26.017916 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02da6bdfe4648e99118631378a37632" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab"
I20260620 02:28:26.018148 2589 raft_consensus.cc:3037] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:26.018168 2310 raft_consensus.cc:3037] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:26.021214 2589 raft_consensus.cc:2454] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 1.
I20260620 02:28:26.021214 2310 raft_consensus.cc:2454] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 1.
I20260620 02:28:26.021724 2376 leader_election.cc:302] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:26.022039 2661 raft_consensus.cc:2781] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:26.022218 2661 raft_consensus.cc:686] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [term 1 LEADER]: Becoming Leader. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Running, Role: LEADER
I20260620 02:28:26.022523 2661 consensus_queue.cc:237] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.024288 2665 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: TxnStatusTablet state changed. Reason: New leader 439aab12911b4b71b04dc8472d20d731. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.024713 2665 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:28:26.025446 2677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:28:26.025748 2677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:28:26.025977 2147 catalog_manager.cc:5495] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 reported cstate change: term changed from 0 to 1, leader changed from <none> to 439aab12911b4b71b04dc8472d20d731 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } health_report { overall_health: UNKNOWN } } }
I20260620 02:28:26.057364 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.051684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.0.97.2:41561)
0620 02:28:26.051895 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:26.051900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:26.051919 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:28:26.052215 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:26.052220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:26.052296 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:26.052458 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:26.052824 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.053726 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.055036 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.055488 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.055645 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.056110 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:26.056655 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:26.056822 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:26.056900 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 02:28:26.057020 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:28:26.057394 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.051595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41561 (local address 127.0.0.1:37010)
0620 02:28:26.051853 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:26.051871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:26.051995 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:26.052486 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:26.052491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:26.052504 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:26.052688 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.052696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.053876 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.053881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.054891 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.054903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.055663 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.055669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.055844 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.055901 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:26.056776 (+ 875us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:26.056798 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:28:26.056884 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:28:26.057067 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:28:26.066833 2310 raft_consensus.cc:1261] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Refusing update from remote peer 439aab12911b4b71b04dc8472d20d731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:26.066846 2589 raft_consensus.cc:1261] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Refusing update from remote peer 439aab12911b4b71b04dc8472d20d731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:26.067983 2665 consensus_queue.cc:1035] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:26.068365 2661 consensus_queue.cc:1035] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:26.086104 2662 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: TxnStatusTablet state changed. Reason: New leader 439aab12911b4b71b04dc8472d20d731. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.086413 2662 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:26.087651 2680 mvcc.cc:205] Tried to move back new op lower bound from 7298754584838635520 to 7298754584670896128. Current Snapshot: MvccSnapshot[applied={T|T < 7298754584838635520}]
I20260620 02:28:26.087972 2681 mvcc.cc:205] Tried to move back new op lower bound from 7298754584838635520 to 7298754584670896128. Current Snapshot: MvccSnapshot[applied={T|T < 7298754584838635520}]
I20260620 02:28:26.090504 2665 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.090778 2665 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:28:26.094022 2662 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.094265 2662 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P a2469b27d35849a980836e9baa0abac0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:26.098347 2660 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: TxnStatusTablet state changed. Reason: New leader 439aab12911b4b71b04dc8472d20d731. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.098651 2660 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:26.099951 2661 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.100283 2661 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P 439aab12911b4b71b04dc8472d20d731: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:28:26.102531 2682 mvcc.cc:205] Tried to move back new op lower bound from 7298754584838635520 to 7298754584670896128. Current Snapshot: MvccSnapshot[applied={T|T < 7298754584838635520}]
I20260620 02:28:26.103277 2660 tablet_replica.cc:439] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } } }
I20260620 02:28:26.103534 2660 tablet_replica.cc:442] T b02da6bdfe4648e99118631378a37632 P e402dd8bc41e44efa166b3bd488602ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:28:26.114331 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.107251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.0.97.2:41561)
0620 02:28:26.107473 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:26.107483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:26.107994 (+ 511us) server_negotiation.cc:407] Connection header received
0620 02:28:26.108173 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:26.108177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:26.108261 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:26.108339 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:26.108831 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.110151 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.112279 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.112863 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.112968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.113498 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:26.113667 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:26.113738 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:26.113901 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 02:28:26.113974 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:28:26.133128 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.125909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42711 (local address 127.0.97.2:41561)
0620 02:28:26.126188 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:26.126196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:26.127045 (+ 849us) server_negotiation.cc:407] Connection header received
0620 02:28:26.127211 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:26.127216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:26.127299 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:26.127402 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:26.127957 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.129155 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.130785 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.131496 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.131626 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.132215 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:26.132368 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:26.132464 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:26.132646 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 02:28:26.132739 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:28:26.133266 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.126019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41561 (local address 127.0.97.2:42711)
0620 02:28:26.126945 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:26.126962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:26.127088 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:26.127430 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:26.127433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:26.127442 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:26.127793 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.127801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.129282 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.129286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.130647 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.130661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.131719 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.131725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.131874 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.131960 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:26.132504 (+ 544us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:26.132515 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:28:26.132600 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 02:28:26.132738 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":3,"thread_start_us":561,"threads_started":1}
I20260620 02:28:26.144512 2310 raft_consensus.cc:1261] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Refusing update from remote peer 439aab12911b4b71b04dc8472d20d731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:26.144587 2589 raft_consensus.cc:1261] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Refusing update from remote peer 439aab12911b4b71b04dc8472d20d731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:26.145221 2693 consensus_queue.cc:1035] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:26.145718 2661 consensus_queue.cc:1035] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:26.208981 2148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:26.209360 2148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:26.213070 2148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Sending DeleteTablet for 3 replicas of tablet 1273c9f87f204936aab2ade9c8646268
I20260620 02:28:26.214305 2565 tablet_service.cc:1514] Processing DeleteTablet for tablet 1273c9f87f204936aab2ade9c8646268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 02:28:26.215385 2698 tablet_replica.cc:330] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: stopping tablet replica
I20260620 02:28:26.215296 2285 tablet_service.cc:1514] Processing DeleteTablet for tablet 1273c9f87f204936aab2ade9c8646268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 02:28:26.215639 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:28:26.216050 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 1273c9f87f204936aab2ade9c8646268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36996
W20260620 02:28:26.216427 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:26.216732 2698 raft_consensus.cc:2229] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:26.217240 2698 raft_consensus.cc:2258] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:26.219302 2698 ts_tablet_manager.cc:1905] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:26.226611 2565 tablet_service.cc:1467] Processing CreateTablet for tablet 223f924aaab747dfa92520941bc72a61 (DEFAULT_TABLE table=test-table [id=7808767aa4224a62b0c348c912222429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:26.227182 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223f924aaab747dfa92520941bc72a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:26.227667 2698 ts_tablet_manager.cc:1918] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:26.227944 2698 log.cc:1196] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/1273c9f87f204936aab2ade9c8646268
I20260620 02:28:26.228658 2698 ts_tablet_manager.cc:1939] T 1273c9f87f204936aab2ade9c8646268 P e402dd8bc41e44efa166b3bd488602ab: Deleting consensus metadata
I20260620 02:28:26.234359 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 223f924aaab747dfa92520941bc72a61 (DEFAULT_TABLE table=test-table [id=7808767aa4224a62b0c348c912222429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:26.234757 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 223f924aaab747dfa92520941bc72a61 (DEFAULT_TABLE table=test-table [id=7808767aa4224a62b0c348c912222429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:26.234946 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223f924aaab747dfa92520941bc72a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:26.235296 2700 tablet_replica.cc:330] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: stopping tablet replica
I20260620 02:28:26.235366 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223f924aaab747dfa92520941bc72a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:26.235870 2700 raft_consensus.cc:2229] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:26.235929 2699 tablet_replica.cc:330] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: stopping tablet replica
I20260620 02:28:26.236379 2699 raft_consensus.cc:2229] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:26.236799 2699 raft_consensus.cc:2258] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:26.237250 2133 catalog_manager.cc:4841] TS e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265): tablet 1273c9f87f204936aab2ade9c8646268 (table test-table [id=4518dfda786d44419dd5db8c6092db8e]) successfully deleted
I20260620 02:28:26.239164 2699 ts_tablet_manager.cc:1905] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:26.240074 2654 tablet_bootstrap.cc:492] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Bootstrap starting.
I20260620 02:28:26.240726 2700 raft_consensus.cc:2258] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:26.241897 2654 tablet_bootstrap.cc:654] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:26.242043 2657 tablet_bootstrap.cc:492] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Bootstrap starting.
I20260620 02:28:26.243815 2657 tablet_bootstrap.cc:654] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:26.246567 2655 tablet_bootstrap.cc:492] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Bootstrap starting.
I20260620 02:28:26.247020 2700 ts_tablet_manager.cc:1905] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:26.248309 2655 tablet_bootstrap.cc:654] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:26.259676 2655 tablet_bootstrap.cc:492] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: No bootstrap required, opened a new log
I20260620 02:28:26.259899 2655 ts_tablet_manager.cc:1397] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260620 02:28:26.260990 2655 raft_consensus.cc:348] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.261467 2655 raft_consensus.cc:374] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:26.261579 2655 raft_consensus.cc:729] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Initialized, Role: FOLLOWER
I20260620 02:28:26.261801 2657 tablet_bootstrap.cc:492] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: No bootstrap required, opened a new log
I20260620 02:28:26.261984 2657 ts_tablet_manager.cc:1397] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.012s
I20260620 02:28:26.261885 2655 consensus_queue.cc:260] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.262748 2657 raft_consensus.cc:348] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.263000 2657 raft_consensus.cc:374] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:26.263105 2657 raft_consensus.cc:729] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2469b27d35849a980836e9baa0abac0, State: Initialized, Role: FOLLOWER
I20260620 02:28:26.263406 2657 consensus_queue.cc:260] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.264066 2657 ts_tablet_manager.cc:1428] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 02:28:26.264688 2700 ts_tablet_manager.cc:1918] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:26.264904 2700 log.cc:1196] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/1273c9f87f204936aab2ade9c8646268
I20260620 02:28:26.265532 2700 ts_tablet_manager.cc:1939] T 1273c9f87f204936aab2ade9c8646268 P 439aab12911b4b71b04dc8472d20d731: Deleting consensus metadata
I20260620 02:28:26.266052 2699 ts_tablet_manager.cc:1918] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:26.266233 2699 log.cc:1196] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/1273c9f87f204936aab2ade9c8646268
I20260620 02:28:26.266800 2699 ts_tablet_manager.cc:1939] T 1273c9f87f204936aab2ade9c8646268 P a2469b27d35849a980836e9baa0abac0: Deleting consensus metadata
I20260620 02:28:26.267937 2132 catalog_manager.cc:4841] TS 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561): tablet 1273c9f87f204936aab2ade9c8646268 (table test-table [id=4518dfda786d44419dd5db8c6092db8e]) successfully deleted
I20260620 02:28:26.268477 2132 catalog_manager.cc:4841] TS a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829): tablet 1273c9f87f204936aab2ade9c8646268 (table test-table [id=4518dfda786d44419dd5db8c6092db8e]) successfully deleted
I20260620 02:28:26.269032 2655 ts_tablet_manager.cc:1428] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260620 02:28:26.269919 2654 tablet_bootstrap.cc:492] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: No bootstrap required, opened a new log
I20260620 02:28:26.270126 2654 ts_tablet_manager.cc:1397] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.005s
I20260620 02:28:26.270988 2654 raft_consensus.cc:348] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.271291 2654 raft_consensus.cc:374] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:26.271435 2654 raft_consensus.cc:729] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Initialized, Role: FOLLOWER
I20260620 02:28:26.271714 2654 consensus_queue.cc:260] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.272302 2654 ts_tablet_manager.cc:1428] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 02:28:26.694557 2697 raft_consensus.cc:482] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:26.694828 2697 raft_consensus.cc:504] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.695256 2662 raft_consensus.cc:482] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:26.695483 2662 raft_consensus.cc:504] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.695708 2697 leader_election.cc:288] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265)
I20260620 02:28:26.696295 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223f924aaab747dfa92520941bc72a61" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0" is_pre_election: true
I20260620 02:28:26.696532 2310 raft_consensus.cc:2454] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 0.
I20260620 02:28:26.697038 2375 leader_election.cc:302] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, a2469b27d35849a980836e9baa0abac0; no voters:
I20260620 02:28:26.697248 2662 leader_election.cc:288] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561), e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265)
I20260620 02:28:26.697326 2697 raft_consensus.cc:2781] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:26.697451 2697 raft_consensus.cc:482] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:26.697544 2697 raft_consensus.cc:3037] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:26.697646 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223f924aaab747dfa92520941bc72a61" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab" is_pre_election: true
I20260620 02:28:26.697873 2589 raft_consensus.cc:2454] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 0.
I20260620 02:28:26.701438 2697 raft_consensus.cc:504] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.703500 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223f924aaab747dfa92520941bc72a61" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab"
I20260620 02:28:26.703787 2589 raft_consensus.cc:3037] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:26.707708 2589 raft_consensus.cc:2454] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 1.
I20260620 02:28:26.708303 2376 leader_election.cc:302] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:26.710602 2697 leader_election.cc:288] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [CANDIDATE]: Term 1 election: Requested vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265)
I20260620 02:28:26.710882 2697 raft_consensus.cc:2781] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:26.711050 2697 raft_consensus.cc:686] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 1 LEADER]: Becoming Leader. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Running, Role: LEADER
I20260620 02:28:26.711411 2697 consensus_queue.cc:237] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:26.721817 2147 catalog_manager.cc:5495] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 reported cstate change: term changed from 0 to 1, leader changed from <none> to 439aab12911b4b71b04dc8472d20d731 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "439aab12911b4b71b04dc8472d20d731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } health_report { overall_health: UNKNOWN } } }
I20260620 02:28:26.723158 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223f924aaab747dfa92520941bc72a61" candidate_uuid: "439aab12911b4b71b04dc8472d20d731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0"
I20260620 02:28:26.723418 2310 raft_consensus.cc:3037] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:26.725788 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.698192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41561 (local address 127.0.97.1:56651)
0620 02:28:26.716402 (+ 18210us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:26.716432 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:26.716635 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:26.717388 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:26.717396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:26.717432 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:26.717848 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.717863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.719886 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.719894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.721990 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.722007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.724813 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.724823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.725055 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.725098 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:28:26.725201 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:28:26.725379 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18033,"thread_start_us":18074,"threads_started":1}
I20260620 02:28:26.726634 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.698718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56651 (local address 127.0.97.2:41561)
0620 02:28:26.715528 (+ 16810us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:26.715537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:26.716783 (+ 1246us) server_negotiation.cc:407] Connection header received
0620 02:28:26.716877 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:26.716885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:26.717024 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:26.717140 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:26.718068 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.719649 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.722194 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.723908 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.724341 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.724489 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:26.726095 (+ 1606us) server_negotiation.cc:299] Negotiation successful
0620 02:28:26.726248 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16643,"thread_start_us":16838,"threads_started":1}
I20260620 02:28:26.727859 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223f924aaab747dfa92520941bc72a61" candidate_uuid: "a2469b27d35849a980836e9baa0abac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439aab12911b4b71b04dc8472d20d731" is_pre_election: true
I20260620 02:28:26.726850 2310 raft_consensus.cc:2454] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439aab12911b4b71b04dc8472d20d731 in term 1.
I20260620 02:28:26.730911 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.699066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60381 (local address 127.0.97.3:43265)
0620 02:28:26.709306 (+ 10240us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:26.709312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:26.722074 (+ 12762us) server_negotiation.cc:407] Connection header received
0620 02:28:26.722738 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:26.722743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:26.722827 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:26.722901 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:26.723830 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.725048 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.727874 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.729240 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.729343 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.729462 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:26.730243 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 02:28:26.730446 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10065,"thread_start_us":9834,"threads_started":1}
I20260620 02:28:26.730938 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:26.698751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43265 (local address 127.0.97.1:60381)
0620 02:28:26.721949 (+ 23198us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:26.721971 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:26.722543 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:26.723194 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:26.723200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:26.723250 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:26.723653 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.723664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.725281 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.725286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.727689 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:26.727705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:26.729851 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:26.729861 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:26.730032 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:26.730072 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:28:26.730179 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:28:26.730326 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23052,"thread_start_us":23344,"threads_started":1}
I20260620 02:28:26.731663 2589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223f924aaab747dfa92520941bc72a61" candidate_uuid: "a2469b27d35849a980836e9baa0abac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e402dd8bc41e44efa166b3bd488602ab" is_pre_election: true
I20260620 02:28:26.732040 2589 raft_consensus.cc:2379] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2469b27d35849a980836e9baa0abac0 in current term 1: Already voted for candidate 439aab12911b4b71b04dc8472d20d731 in this term.
I20260620 02:28:26.732563 2236 leader_election.cc:302] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2469b27d35849a980836e9baa0abac0; no voters: 439aab12911b4b71b04dc8472d20d731, e402dd8bc41e44efa166b3bd488602ab
I20260620 02:28:26.732977 2662 raft_consensus.cc:2726] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 02:28:26.861872 2588 raft_consensus.cc:1261] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Refusing update from remote peer 439aab12911b4b71b04dc8472d20d731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:26.861956 2309 raft_consensus.cc:1261] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Refusing update from remote peer 439aab12911b4b71b04dc8472d20d731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:26.862712 2719 consensus_queue.cc:1035] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:26.863324 2697 consensus_queue.cc:1035] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:28:26.890888 2712 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:28:28.492278 2148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:28.492643 2148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:28.498374 2148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Sending DeleteTablet for 3 replicas of tablet 223f924aaab747dfa92520941bc72a61
I20260620 02:28:28.499502 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 223f924aaab747dfa92520941bc72a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 02:28:28.499660 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 223f924aaab747dfa92520941bc72a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36996
I20260620 02:28:28.499696 2565 tablet_service.cc:1514] Processing DeleteTablet for tablet 223f924aaab747dfa92520941bc72a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 02:28:28.514050 2744 tablet_replica.cc:330] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: stopping tablet replica
I20260620 02:28:28.514400 2744 raft_consensus.cc:2229] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:28.514847 2744 raft_consensus.cc:2258] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:28.516849 2744 ts_tablet_manager.cc:1905] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:28.525213 2744 ts_tablet_manager.cc:1918] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:28:28.525437 2744 log.cc:1196] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/223f924aaab747dfa92520941bc72a61
I20260620 02:28:28.526489 2746 tablet_replica.cc:330] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: stopping tablet replica
I20260620 02:28:28.526787 2746 raft_consensus.cc:2229] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:28.527817 2744 ts_tablet_manager.cc:1939] T 223f924aaab747dfa92520941bc72a61 P a2469b27d35849a980836e9baa0abac0: Deleting consensus metadata
I20260620 02:28:28.527813 2745 tablet_replica.cc:330] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: stopping tablet replica
I20260620 02:28:28.528081 2745 raft_consensus.cc:2229] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:28.528646 2745 raft_consensus.cc:2258] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:28.529328 2132 catalog_manager.cc:4841] TS a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829): tablet 223f924aaab747dfa92520941bc72a61 (table test-table [id=7808767aa4224a62b0c348c912222429]) successfully deleted
I20260620 02:28:28.532130 2746 raft_consensus.cc:2258] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:28.533350 2746 ts_tablet_manager.cc:1905] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:28.533365 2745 ts_tablet_manager.cc:1905] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:28.541468 2745 ts_tablet_manager.cc:1918] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:28:28.541752 2745 log.cc:1196] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/223f924aaab747dfa92520941bc72a61
I20260620 02:28:28.541754 2746 ts_tablet_manager.cc:1918] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:28:28.541923 2746 log.cc:1196] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/223f924aaab747dfa92520941bc72a61
I20260620 02:28:28.542497 2746 ts_tablet_manager.cc:1939] T 223f924aaab747dfa92520941bc72a61 P e402dd8bc41e44efa166b3bd488602ab: Deleting consensus metadata
I20260620 02:28:28.542514 2745 ts_tablet_manager.cc:1939] T 223f924aaab747dfa92520941bc72a61 P 439aab12911b4b71b04dc8472d20d731: Deleting consensus metadata
I20260620 02:28:28.543771 2133 catalog_manager.cc:4841] TS e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265): tablet 223f924aaab747dfa92520941bc72a61 (table test-table [id=7808767aa4224a62b0c348c912222429]) successfully deleted
I20260620 02:28:28.543771 2132 catalog_manager.cc:4841] TS 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561): tablet 223f924aaab747dfa92520941bc72a61 (table test-table [id=7808767aa4224a62b0c348c912222429]) successfully deleted
Jun 20 02:28:28 dist-test-slave-r706 krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:28:28 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:28.593784 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.573120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.0.97.62:38643)
0620 02:28:28.574170 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:28.574175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:28.574207 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:28:28.574263 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:28.574266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:28.574330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:28.574423 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:28.576123 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.577183 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.578686 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.579219 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.579322 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.587329 (+ 8007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:28.587337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:28.587341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:28.587382 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:28.590979 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:28.591735 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:28.591739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:28.591742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:28.591920 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:28.592268 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:28.592271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:28.592273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:28.592568 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:28.592784 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:28.593088 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 02:28:28.593302 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":640,"threads_started":1}
I20260620 02:28:28.598012 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:28.598628 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:28.605612 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 43e838682fc74728b4c13ac426577977 (DEFAULT_TABLE table=test-table [id=08d3768d16d845f8a81e269760d7febb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:28.605834 2565 tablet_service.cc:1467] Processing CreateTablet for tablet 43e838682fc74728b4c13ac426577977 (DEFAULT_TABLE table=test-table [id=08d3768d16d845f8a81e269760d7febb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:28.605837 2424 tablet_service.cc:1467] Processing CreateTablet for tablet 43e838682fc74728b4c13ac426577977 (DEFAULT_TABLE table=test-table [id=08d3768d16d845f8a81e269760d7febb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:28.606205 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e838682fc74728b4c13ac426577977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:28.606242 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e838682fc74728b4c13ac426577977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:28.606513 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e838682fc74728b4c13ac426577977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:28.613425 2755 tablet_bootstrap.cc:492] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Bootstrap starting.
I20260620 02:28:28.616144 2755 tablet_bootstrap.cc:654] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:28.616966 2756 tablet_bootstrap.cc:492] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Bootstrap starting.
I20260620 02:28:28.617991 2757 tablet_bootstrap.cc:492] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Bootstrap starting.
I20260620 02:28:28.619900 2756 tablet_bootstrap.cc:654] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:28.620793 2757 tablet_bootstrap.cc:654] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:28.621693 2755 tablet_bootstrap.cc:492] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: No bootstrap required, opened a new log
I20260620 02:28:28.621946 2755 ts_tablet_manager.cc:1397] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 02:28:28.623323 2755 raft_consensus.cc:348] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.623593 2756 tablet_bootstrap.cc:492] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: No bootstrap required, opened a new log
I20260620 02:28:28.623641 2755 raft_consensus.cc:374] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:28.623798 2755 raft_consensus.cc:729] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Initialized, Role: FOLLOWER
I20260620 02:28:28.623798 2756 ts_tablet_manager.cc:1397] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 02:28:28.624127 2755 consensus_queue.cc:260] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.624329 2757 tablet_bootstrap.cc:492] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: No bootstrap required, opened a new log
I20260620 02:28:28.624522 2757 ts_tablet_manager.cc:1397] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 02:28:28.625216 2756 raft_consensus.cc:348] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.625600 2756 raft_consensus.cc:374] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:28.625944 2756 raft_consensus.cc:729] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Initialized, Role: FOLLOWER
I20260620 02:28:28.625741 2757 raft_consensus.cc:348] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.626103 2757 raft_consensus.cc:374] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:28.626271 2757 raft_consensus.cc:729] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2469b27d35849a980836e9baa0abac0, State: Initialized, Role: FOLLOWER
I20260620 02:28:28.626356 2756 consensus_queue.cc:260] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.626663 2757 consensus_queue.cc:260] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.627231 2756 ts_tablet_manager.cc:1428] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 02:28:28.628371 2755 ts_tablet_manager.cc:1428] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260620 02:28:28.630100 2757 ts_tablet_manager.cc:1428] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260620 02:28:28.632488 2761 raft_consensus.cc:482] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:28.632862 2761 raft_consensus.cc:504] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.635104 2761 leader_election.cc:288] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561)
I20260620 02:28:28.646111 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.635467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39829 (local address 127.0.97.3:39535)
0620 02:28:28.636495 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:28.636511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:28.636640 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:28.637111 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:28.637123 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:28.637164 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:28.638123 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.638137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.640320 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.640326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.643117 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.643133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.645126 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.645132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.645289 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.645316 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:28:28.645397 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 02:28:28.645495 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":33,"thread_start_us":738,"threads_started":1}
I20260620 02:28:28.646147 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.635307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39535 (local address 127.0.97.1:39829)
0620 02:28:28.636371 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:28.636386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:28.636601 (+ 215us) server_negotiation.cc:407] Connection header received
0620 02:28:28.636748 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:28.636756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:28.636867 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:28.636990 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:28.638401 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.640103 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.643341 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.644810 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.644982 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.645158 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:28.645450 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:28:28.645640 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":618,"threads_started":1}
I20260620 02:28:28.647120 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.636678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41561 (local address 127.0.97.3:36065)
0620 02:28:28.637927 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:28.637961 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:28.638140 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:28.638582 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:28.638585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:28.638608 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:28.639160 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.639169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.641471 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.641477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.643865 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.643877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.645986 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.646000 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.646266 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.646334 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 02:28:28.646423 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:28:28.646583 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":1,"thread_start_us":726,"threads_started":1}
I20260620 02:28:28.647073 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e838682fc74728b4c13ac426577977" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0" is_pre_election: true
I20260620 02:28:28.647452 2309 raft_consensus.cc:2454] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 0.
I20260620 02:28:28.648046 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.637182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36065 (local address 127.0.97.2:41561)
0620 02:28:28.638078 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:28.638083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:28.638111 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:28:28.638314 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:28.638320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:28.638410 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:28.638561 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:28.639393 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.641318 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.644059 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.645668 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.645820 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.646019 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:28.647374 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0620 02:28:28.647515 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":541,"threads_started":1}
I20260620 02:28:28.648072 2515 leader_election.cc:302] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2469b27d35849a980836e9baa0abac0, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:28.648607 2761 raft_consensus.cc:2781] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:28.648653 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e838682fc74728b4c13ac426577977" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439aab12911b4b71b04dc8472d20d731" is_pre_election: true
I20260620 02:28:28.648772 2761 raft_consensus.cc:482] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:28.648947 2761 raft_consensus.cc:3037] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:28.649117 2450 raft_consensus.cc:2454] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 0.
I20260620 02:28:28.652240 2761 raft_consensus.cc:504] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.652904 2761 leader_election.cc:288] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 election: Requested vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561)
I20260620 02:28:28.653685 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e838682fc74728b4c13ac426577977" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0"
I20260620 02:28:28.653817 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e838682fc74728b4c13ac426577977" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439aab12911b4b71b04dc8472d20d731"
I20260620 02:28:28.653970 2309 raft_consensus.cc:3037] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:28.654158 2450 raft_consensus.cc:3037] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:28.656529 2309 raft_consensus.cc:2454] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 1.
I20260620 02:28:28.656957 2450 raft_consensus.cc:2454] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 1.
I20260620 02:28:28.657088 2515 leader_election.cc:302] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2469b27d35849a980836e9baa0abac0, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:28.657482 2761 raft_consensus.cc:2781] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:28.658272 2761 raft_consensus.cc:686] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 1 LEADER]: Becoming Leader. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Running, Role: LEADER
I20260620 02:28:28.658746 2761 consensus_queue.cc:237] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.662976 2146 catalog_manager.cc:5495] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab reported cstate change: term changed from 0 to 1, leader changed from <none> to e402dd8bc41e44efa166b3bd488602ab (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e402dd8bc41e44efa166b3bd488602ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } health_report { overall_health: HEALTHY } } }
I20260620 02:28:28.697518 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.690163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.0.97.3:43265)
0620 02:28:28.691156 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:28.691160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:28.691187 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:28:28.691276 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:28.691279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:28.691356 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:28.691523 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:28.692070 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.693140 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.694806 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.695369 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.695470 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.696038 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:28.696684 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:28.696800 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:28.696978 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 02:28:28.697116 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":652,"threads_started":1}
I20260620 02:28:28.717103 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.710156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59851 (local address 127.0.97.2:41561)
0620 02:28:28.710364 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:28.710369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:28.711311 (+ 942us) server_negotiation.cc:407] Connection header received
0620 02:28:28.711481 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:28.711485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:28.711561 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:28.711700 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:28.712396 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.713386 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.715065 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.715632 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.715753 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.716283 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:28.716420 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:28.716505 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:28.716683 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 02:28:28.716754 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:28:28.717402 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.710213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41561 (local address 127.0.97.3:59851)
0620 02:28:28.711186 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:28.711202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:28.711352 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:28.711675 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:28.711678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:28.711687 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:28.712237 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.712245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.713527 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.713531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.714914 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.714926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.715831 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.715835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.715987 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.716045 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:28.716547 (+ 502us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:28.716560 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:28:28.716649 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:28:28.716766 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":574,"threads_started":1}
I20260620 02:28:28.727566 2309 raft_consensus.cc:1261] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Refusing update from remote peer e402dd8bc41e44efa166b3bd488602ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:28.727732 2450 raft_consensus.cc:1261] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Refusing update from remote peer e402dd8bc41e44efa166b3bd488602ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:28.728283 2767 consensus_queue.cc:1035] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:28.728909 2761 consensus_queue.cc:1035] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:28.761040 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.752534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33599 (local address 127.0.97.3:43265)
0620 02:28:28.752697 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:28.752701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:28.753658 (+ 957us) server_negotiation.cc:407] Connection header received
0620 02:28:28.753887 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:28.753893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:28.753990 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:28.754068 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:28.755371 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.756472 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.758570 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.759388 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.759552 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.760119 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:28.760255 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:28.760348 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:28.760500 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:28:28.760579 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 02:28:28.761214 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:28.752532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43265 (local address 127.0.97.2:33599)
0620 02:28:28.753536 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:28.753556 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:28.753731 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:28.754450 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:28.754453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:28.754463 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:28:28.755129 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.755139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.756613 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.756617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.758410 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:28.758429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:28.759638 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:28.759643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:28.759877 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:28.759917 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:28:28.760369 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:28:28.760380 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:28:28.760479 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:28:28.760617 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":606,"threads_started":1}
I20260620 02:28:28.797972 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:28.798317 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:28.801837 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Sending DeleteTablet for 3 replicas of tablet 43e838682fc74728b4c13ac426577977
I20260620 02:28:28.802975 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e838682fc74728b4c13ac426577977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 02:28:28.803369 2744 tablet_replica.cc:330] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: stopping tablet replica
I20260620 02:28:28.803421 2565 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e838682fc74728b4c13ac426577977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 02:28:28.803512 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e838682fc74728b4c13ac426577977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36996
I20260620 02:28:28.803651 2744 raft_consensus.cc:2229] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:28.803828 2746 tablet_replica.cc:330] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: stopping tablet replica
I20260620 02:28:28.803857 2745 tablet_replica.cc:330] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: stopping tablet replica
I20260620 02:28:28.803939 2744 raft_consensus.cc:2258] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:28.804018 2746 raft_consensus.cc:2229] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:28.804150 2745 raft_consensus.cc:2229] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:28.804391 2745 raft_consensus.cc:2258] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:28.804626 2746 raft_consensus.cc:2258] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:28.805758 2745 ts_tablet_manager.cc:1905] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:28.805841 2746 ts_tablet_manager.cc:1905] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:28.805943 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:28.806682 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:28.809834 2744 ts_tablet_manager.cc:1905] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:28.813939 2745 ts_tablet_manager.cc:1918] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:28.814328 2745 log.cc:1196] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/43e838682fc74728b4c13ac426577977
I20260620 02:28:28.815052 2745 ts_tablet_manager.cc:1939] T 43e838682fc74728b4c13ac426577977 P 439aab12911b4b71b04dc8472d20d731: Deleting consensus metadata
I20260620 02:28:28.816581 2132 catalog_manager.cc:4841] TS 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561): tablet 43e838682fc74728b4c13ac426577977 (table test-table [id=08d3768d16d845f8a81e269760d7febb]) successfully deleted
I20260620 02:28:28.817762 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 0f987cae1de544e39c0dc56160e2e82d (DEFAULT_TABLE table=test-table [id=118ab683416e4e24a2dac0b49b7d93a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:28.818337 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f987cae1de544e39c0dc56160e2e82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:28.822908 2744 ts_tablet_manager.cc:1918] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:28.823192 2744 log.cc:1196] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/43e838682fc74728b4c13ac426577977
I20260620 02:28:28.823666 2746 ts_tablet_manager.cc:1918] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:28:28.823758 2744 ts_tablet_manager.cc:1939] T 43e838682fc74728b4c13ac426577977 P a2469b27d35849a980836e9baa0abac0: Deleting consensus metadata
I20260620 02:28:28.823994 2746 log.cc:1196] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/43e838682fc74728b4c13ac426577977
I20260620 02:28:28.824976 2132 catalog_manager.cc:4841] TS a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829): tablet 43e838682fc74728b4c13ac426577977 (table test-table [id=08d3768d16d845f8a81e269760d7febb]) successfully deleted
I20260620 02:28:28.825198 2424 tablet_service.cc:1467] Processing CreateTablet for tablet 0f987cae1de544e39c0dc56160e2e82d (DEFAULT_TABLE table=test-table [id=118ab683416e4e24a2dac0b49b7d93a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:28.825615 2565 tablet_service.cc:1467] Processing CreateTablet for tablet 0f987cae1de544e39c0dc56160e2e82d (DEFAULT_TABLE table=test-table [id=118ab683416e4e24a2dac0b49b7d93a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:28.825829 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f987cae1de544e39c0dc56160e2e82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:28.825840 2746 ts_tablet_manager.cc:1939] T 43e838682fc74728b4c13ac426577977 P e402dd8bc41e44efa166b3bd488602ab: Deleting consensus metadata
I20260620 02:28:28.826262 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f987cae1de544e39c0dc56160e2e82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:28.827587 2757 tablet_bootstrap.cc:492] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Bootstrap starting.
I20260620 02:28:28.829818 2133 catalog_manager.cc:4841] TS e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265): tablet 43e838682fc74728b4c13ac426577977 (table test-table [id=08d3768d16d845f8a81e269760d7febb]) successfully deleted
I20260620 02:28:28.830507 2757 tablet_bootstrap.cc:654] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:28.835711 2755 tablet_bootstrap.cc:492] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Bootstrap starting.
I20260620 02:28:28.835744 2756 tablet_bootstrap.cc:492] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Bootstrap starting.
I20260620 02:28:28.838562 2755 tablet_bootstrap.cc:654] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:28.838976 2756 tablet_bootstrap.cc:654] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:28.842063 2757 tablet_bootstrap.cc:492] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: No bootstrap required, opened a new log
I20260620 02:28:28.842370 2757 ts_tablet_manager.cc:1397] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260620 02:28:28.843700 2757 raft_consensus.cc:348] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.844132 2757 raft_consensus.cc:374] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:28.844282 2757 raft_consensus.cc:729] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2469b27d35849a980836e9baa0abac0, State: Initialized, Role: FOLLOWER
I20260620 02:28:28.844722 2757 consensus_queue.cc:260] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.846678 2757 ts_tablet_manager.cc:1428] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 02:28:28.848942 2756 tablet_bootstrap.cc:492] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: No bootstrap required, opened a new log
I20260620 02:28:28.849251 2756 ts_tablet_manager.cc:1397] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260620 02:28:28.850438 2755 tablet_bootstrap.cc:492] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: No bootstrap required, opened a new log
I20260620 02:28:28.850509 2756 raft_consensus.cc:348] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.850750 2755 ts_tablet_manager.cc:1397] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20260620 02:28:28.850876 2756 raft_consensus.cc:374] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:28.851028 2756 raft_consensus.cc:729] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439aab12911b4b71b04dc8472d20d731, State: Initialized, Role: FOLLOWER
I20260620 02:28:28.851402 2756 consensus_queue.cc:260] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.851846 2755 raft_consensus.cc:348] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.852195 2755 raft_consensus.cc:374] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:28.852319 2756 ts_tablet_manager.cc:1428] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:28.852339 2755 raft_consensus.cc:729] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Initialized, Role: FOLLOWER
I20260620 02:28:28.852662 2755 consensus_queue.cc:260] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.853366 2755 ts_tablet_manager.cc:1428] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:28:28.902070 2761 raft_consensus.cc:482] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:28.902333 2761 raft_consensus.cc:504] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.903281 2761 leader_election.cc:288] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561)
I20260620 02:28:28.903915 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f987cae1de544e39c0dc56160e2e82d" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0" is_pre_election: true
I20260620 02:28:28.904196 2309 raft_consensus.cc:2454] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 0.
I20260620 02:28:28.904117 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f987cae1de544e39c0dc56160e2e82d" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439aab12911b4b71b04dc8472d20d731" is_pre_election: true
I20260620 02:28:28.904508 2450 raft_consensus.cc:2454] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 0.
I20260620 02:28:28.904847 2515 leader_election.cc:302] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2469b27d35849a980836e9baa0abac0, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:28.905532 2761 raft_consensus.cc:2781] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:28.905771 2761 raft_consensus.cc:482] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:28.905929 2761 raft_consensus.cc:3037] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:28.908790 2761 raft_consensus.cc:504] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.909469 2761 leader_election.cc:288] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 election: Requested vote from peers a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829), 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561)
I20260620 02:28:28.910105 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f987cae1de544e39c0dc56160e2e82d" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2469b27d35849a980836e9baa0abac0"
I20260620 02:28:28.910164 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f987cae1de544e39c0dc56160e2e82d" candidate_uuid: "e402dd8bc41e44efa166b3bd488602ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439aab12911b4b71b04dc8472d20d731"
I20260620 02:28:28.910511 2450 raft_consensus.cc:3037] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:28.910496 2309 raft_consensus.cc:3037] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:28.914445 2450 raft_consensus.cc:2454] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 1.
I20260620 02:28:28.914445 2309 raft_consensus.cc:2454] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e402dd8bc41e44efa166b3bd488602ab in term 1.
I20260620 02:28:28.914955 2515 leader_election.cc:302] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439aab12911b4b71b04dc8472d20d731, e402dd8bc41e44efa166b3bd488602ab; no voters:
I20260620 02:28:28.915470 2761 raft_consensus.cc:2781] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:28.915756 2761 raft_consensus.cc:686] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 1 LEADER]: Becoming Leader. State: Replica: e402dd8bc41e44efa166b3bd488602ab, State: Running, Role: LEADER
I20260620 02:28:28.916152 2761 consensus_queue.cc:237] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } }
I20260620 02:28:28.920022 2146 catalog_manager.cc:5495] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab reported cstate change: term changed from 0 to 1, leader changed from <none> to e402dd8bc41e44efa166b3bd488602ab (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e402dd8bc41e44efa166b3bd488602ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e402dd8bc41e44efa166b3bd488602ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43265 } health_report { overall_health: HEALTHY } } }
I20260620 02:28:28.953186 2450 raft_consensus.cc:1261] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Refusing update from remote peer e402dd8bc41e44efa166b3bd488602ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:28.953972 2761 consensus_queue.cc:1035] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "439aab12911b4b71b04dc8472d20d731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:28.959311 2309 raft_consensus.cc:1261] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Refusing update from remote peer e402dd8bc41e44efa166b3bd488602ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:28.966243 2761 consensus_queue.cc:1035] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2469b27d35849a980836e9baa0abac0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260620 02:28:29.010028 2727 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:28:30.618393 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:30.618731 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:30.624205 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e883322978fb4e3e82c3d30791e60b71: Sending DeleteTablet for 3 replicas of tablet 0f987cae1de544e39c0dc56160e2e82d
I20260620 02:28:30.625564 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f987cae1de544e39c0dc56160e2e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36996
I20260620 02:28:30.625581 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f987cae1de544e39c0dc56160e2e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 02:28:30.625731 2565 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f987cae1de544e39c0dc56160e2e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48822
I20260620 02:28:30.626408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2219
I20260620 02:28:30.637665 2808 tablet_replica.cc:330] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: stopping tablet replica
I20260620 02:28:30.638101 2808 raft_consensus.cc:2229] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:30.638602 2808 raft_consensus.cc:2258] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:30.639781 2807 tablet_replica.cc:330] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: stopping tablet replica
I20260620 02:28:30.641115 2807 raft_consensus.cc:2229] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:30.641201 2808 ts_tablet_manager.cc:1905] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:30.641448 2807 raft_consensus.cc:2258] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:30.642110 2809 tablet_replica.cc:330] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: stopping tablet replica
I20260620 02:28:30.642354 2809 raft_consensus.cc:2229] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:30.642858 2809 raft_consensus.cc:2258] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:30.643306 2807 ts_tablet_manager.cc:1905] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:30.644560 2809 ts_tablet_manager.cc:1905] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:30.651777 2808 ts_tablet_manager.cc:1918] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:28:30.652004 2808 log.cc:1196] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/0f987cae1de544e39c0dc56160e2e82d
I20260620 02:28:30.652920 2808 ts_tablet_manager.cc:1939] T 0f987cae1de544e39c0dc56160e2e82d P a2469b27d35849a980836e9baa0abac0: Deleting consensus metadata
I20260620 02:28:30.653254 2807 ts_tablet_manager.cc:1918] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:28:30.653363 2809 ts_tablet_manager.cc:1918] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:28:30.653606 2807 log.cc:1196] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/0f987cae1de544e39c0dc56160e2e82d
I20260620 02:28:30.653692 2809 log.cc:1196] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/0f987cae1de544e39c0dc56160e2e82d
I20260620 02:28:30.654423 2807 ts_tablet_manager.cc:1939] T 0f987cae1de544e39c0dc56160e2e82d P 439aab12911b4b71b04dc8472d20d731: Deleting consensus metadata
I20260620 02:28:30.654536 2809 ts_tablet_manager.cc:1939] T 0f987cae1de544e39c0dc56160e2e82d P e402dd8bc41e44efa166b3bd488602ab: Deleting consensus metadata
I20260620 02:28:30.654767 2132 catalog_manager.cc:4841] TS a2469b27d35849a980836e9baa0abac0 (127.0.97.1:39829): tablet 0f987cae1de544e39c0dc56160e2e82d (table test-table [id=118ab683416e4e24a2dac0b49b7d93a2]) successfully deleted
Jun 20 02:28:30 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:30.655830 2132 catalog_manager.cc:4841] TS 439aab12911b4b71b04dc8472d20d731 (127.0.97.2:41561): tablet 0f987cae1de544e39c0dc56160e2e82d (table test-table [id=118ab683416e4e24a2dac0b49b7d93a2]) successfully deleted
I20260620 02:28:30.655925 2133 catalog_manager.cc:4841] TS e402dd8bc41e44efa166b3bd488602ab (127.0.97.3:43265): tablet 0f987cae1de544e39c0dc56160e2e82d (table test-table [id=118ab683416e4e24a2dac0b49b7d93a2]) successfully deleted
I20260620 02:28:30.664387 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:30.629152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.97.1:39829)
0620 02:28:30.641020 (+ 11868us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:30.641026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:30.641070 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:28:30.641168 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:30.641173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:30.641277 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:30.641467 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:30.644291 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:30.645851 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:30.648122 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:30.649033 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:30.649212 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:30.656986 (+ 7774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:30.657064 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:30.657092 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:30.657152 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:30.660981 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:30.661951 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:30.661960 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:30.661962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:30.662207 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:30.662615 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:30.662618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:30.662620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:30.663046 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:30.663418 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:30.663551 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 02:28:30.663918 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11687,"thread_start_us":11559,"threads_started":1}
W20260620 02:28:30.665455 2350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40364
W20260620 02:28:30.666391 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:30.666693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 2219
I20260620 02:28:30.686302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2359
Jun 20 02:28:30 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:30.704166 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:30.687355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.97.2:41561)
0620 02:28:30.688052 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:30.688057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:30.688073 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:28:30.688146 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:30.688150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:30.688220 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:30.688353 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:30.689900 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:30.690921 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:30.692238 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:30.692685 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:30.692830 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:30.697669 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:30.697697 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:30.697713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:30.697761 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:30.700868 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:30.701808 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:30.701817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:30.701819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:30.702004 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:30.702385 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:30.702390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:30.702394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:30.702874 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:30.703191 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:30.703535 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 02:28:30.703776 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":412,"threads_started":1}
W20260620 02:28:30.704938 2490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36410
W20260620 02:28:30.705891 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2359: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:30.706084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 2359
I20260620 02:28:30.733359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2499
Jun 20 02:28:30 dist-test-slave-r706 krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:30.749814 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:30.734179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.0.97.3:43265)
0620 02:28:30.734752 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:30.734756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:30.734772 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:28:30.734838 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:30.734842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:30.734917 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:30.735015 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:30.736367 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:30.737271 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:30.738469 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:30.738900 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:30.738991 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:30.743508 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:30.743525 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:30.743539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:30.743578 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:30.746794 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:30.747454 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:30.747461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:30.747464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:30.747599 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:30.747971 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:30.747973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:30.747975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:30.748609 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:30.748873 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:30.749317 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 02:28:30.749499 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":317,"threads_started":1}
W20260620 02:28:30.750645 2630 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39818
W20260620 02:28:30.751603 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:30.751791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 2499
I20260620 02:28:30.775377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2116
I20260620 02:28:30.776695 2188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:30.912015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 2116
2026-06-20T02:28:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9148 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:31 dist-test-slave-r706 krb5kdc[2817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:31 dist-test-slave-r706 krb5kdc[2817](info): set up 2 sockets
Jun 20 02:28:31 dist-test-slave-r706 krb5kdc[2817](info): commencing operation
krb5kdc: starting...
W20260620 02:28:32.025518 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:32 dist-test-slave-r706 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:32.237499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43017
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:32.564636 2830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:32.565066 2830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:32.565235 2830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:32.572752 2830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:32.572956 2830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:32.573112 2830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:32.573225 2830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:32.573294 2830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:32.583490 2830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:32.584977 2830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:32.586642 2830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:32.599347 2838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:32.599399 2836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:32.599432 2835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:32.600188 2830 server_base.cc:1034] running on GCE node
I20260620 02:28:32.601186 2830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:32.602784 2830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:32.604168 2830 hybrid_clock.cc:648] HybridClock initialized: now 1781922512603972 us; error 137 us; skew 500 ppm
I20260620 02:28:32.604879 2830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:32 dist-test-slave-r706 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:32.609324 2830 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:32.611171 2830 webserver.cc:460] Webserver started at http://127.0.97.62:36961/ using document root <none> and password file <none>
I20260620 02:28:32.611939 2830 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:32.612063 2830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:32.612450 2830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:32.615713 2830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "cff856c85b8e4f3fa59e5321648c3692"
format_stamp: "Formatted at 2026-06-20 02:28:32 on dist-test-slave-r706"
server_key: "3eedb7e70c5501e7036bab1a87e65c53"
server_key_iv: "a2c8880bff966bb8567a086fa7813fbb"
server_key_version: "encryptionkey@0"
I20260620 02:28:32.616562 2830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "cff856c85b8e4f3fa59e5321648c3692"
format_stamp: "Formatted at 2026-06-20 02:28:32 on dist-test-slave-r706"
server_key: "3eedb7e70c5501e7036bab1a87e65c53"
server_key_iv: "a2c8880bff966bb8567a086fa7813fbb"
server_key_version: "encryptionkey@0"
I20260620 02:28:32.622318 2830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:28:32.626123 2845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:32.627645 2830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:28:32.627871 2830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "cff856c85b8e4f3fa59e5321648c3692"
format_stamp: "Formatted at 2026-06-20 02:28:32 on dist-test-slave-r706"
server_key: "3eedb7e70c5501e7036bab1a87e65c53"
server_key_iv: "a2c8880bff966bb8567a086fa7813fbb"
server_key_version: "encryptionkey@0"
I20260620 02:28:32.628100 2830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:32.676123 2830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:32.681066 2830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:32.681427 2830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:32.703843 2830 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43017
I20260620 02:28:32.703881 2896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43017 every 8 connection(s)
I20260620 02:28:32.705364 2830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:32.708568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 2830
I20260620 02:28:32.708868 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:32.709636 388 external_mini_cluster.cc:1428] Setting key 14c79dcd267f2bcd29418130adcc7679
I20260620 02:28:32.709980 2897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:32.726756 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692: Bootstrap starting.
Jun 20 02:28:32 dist-test-slave-r706 krb5kdc[2817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:32.731020 2897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:32.732365 2897 log.cc:824] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:32.735567 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692: No bootstrap required, opened a new log
I20260620 02:28:32.737278 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:32.712002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.0.97.62:43017)
0620 02:28:32.712925 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:32.712947 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:32.713032 (+ 85us) server_negotiation.cc:407] Connection header received
0620 02:28:32.715826 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:32.715863 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:32.716514 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:32.717104 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:32.718796 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:32.720819 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:32.722424 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:32.723689 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:32.723919 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:32.728673 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:32.728683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:32.728702 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:32.728750 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:32.732545 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:32.733668 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:32.733676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:32.733682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:32.733857 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:32.734284 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:32.734289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:32.734291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:32.734822 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:32.735143 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:32.735617 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 02:28:32.736082 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":381,"threads_started":1}
I20260620 02:28:32.744469 2897 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff856c85b8e4f3fa59e5321648c3692" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43017 } }
I20260620 02:28:32.744920 2897 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:32.745062 2897 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff856c85b8e4f3fa59e5321648c3692, State: Initialized, Role: FOLLOWER
I20260620 02:28:32.745780 2897 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff856c85b8e4f3fa59e5321648c3692" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43017 } }
I20260620 02:28:32.746093 2897 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:32.746279 2897 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:32.746496 2897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:32.748832 2897 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff856c85b8e4f3fa59e5321648c3692" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43017 } }
I20260620 02:28:32.749428 2897 leader_election.cc:302] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff856c85b8e4f3fa59e5321648c3692; no voters:
I20260620 02:28:32.750118 2897 leader_election.cc:288] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:32.750288 2902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:32.751336 2902 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [term 1 LEADER]: Becoming Leader. State: Replica: cff856c85b8e4f3fa59e5321648c3692, State: Running, Role: LEADER
I20260620 02:28:32.751962 2902 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff856c85b8e4f3fa59e5321648c3692" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43017 } }
I20260620 02:28:32.752573 2897 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:32.755429 2903 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff856c85b8e4f3fa59e5321648c3692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff856c85b8e4f3fa59e5321648c3692" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43017 } } }
I20260620 02:28:32.755527 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff856c85b8e4f3fa59e5321648c3692. Latest consensus state: current_term: 1 leader_uuid: "cff856c85b8e4f3fa59e5321648c3692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff856c85b8e4f3fa59e5321648c3692" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43017 } } }
I20260620 02:28:32.755988 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:32.755873 2903 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:32.757084 2910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:32.763180 2910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:32.772797 2910 catalog_manager.cc:1347] Generated new cluster ID: cff7030bf8a74c9d816d5050a5ebd6b9
I20260620 02:28:32.772954 2910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:32.786542 2910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:32.788460 2910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:32.812990 2910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff856c85b8e4f3fa59e5321648c3692: Generated new TSK 0
I20260620 02:28:32.813949 2910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:32.872589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43017
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:33.222347 2923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:33.222740 2923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:33.222914 2923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:33.230414 2923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:33.230593 2923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:33.230785 2923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:33.241240 2923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:33.242609 2923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:33.244009 2923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:33.258335 2929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:33.258347 2928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:33.259209 2931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:33.259188 2923 server_base.cc:1034] running on GCE node
I20260620 02:28:33.259969 2923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:33.261077 2923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:33.262329 2923 hybrid_clock.cc:648] HybridClock initialized: now 1781922513262277 us; error 44 us; skew 500 ppm
I20260620 02:28:33.262686 2923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:33 dist-test-slave-r706 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:33.266026 2923 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:33.267572 2923 webserver.cc:460] Webserver started at http://127.0.97.1:40817/ using document root <none> and password file <none>
I20260620 02:28:33.268296 2923 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:33.268448 2923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:33.268778 2923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:33.271823 2923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "63729287727f4b78afcaed4cf89823d4"
format_stamp: "Formatted at 2026-06-20 02:28:33 on dist-test-slave-r706"
server_key: "6a95ba0c90d6196539ca9e77ae19f51c"
server_key_iv: "6cb1c038e248f0a4c21fd9f22f619aed"
server_key_version: "encryptionkey@0"
I20260620 02:28:33.272629 2923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "63729287727f4b78afcaed4cf89823d4"
format_stamp: "Formatted at 2026-06-20 02:28:33 on dist-test-slave-r706"
server_key: "6a95ba0c90d6196539ca9e77ae19f51c"
server_key_iv: "6cb1c038e248f0a4c21fd9f22f619aed"
server_key_version: "encryptionkey@0"
I20260620 02:28:33.278139 2923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:28:33.281826 2938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:33.283239 2923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:33.283429 2923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "63729287727f4b78afcaed4cf89823d4"
format_stamp: "Formatted at 2026-06-20 02:28:33 on dist-test-slave-r706"
server_key: "6a95ba0c90d6196539ca9e77ae19f51c"
server_key_iv: "6cb1c038e248f0a4c21fd9f22f619aed"
server_key_version: "encryptionkey@0"
I20260620 02:28:33.283671 2923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:33.314481 2923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:33.319495 2923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:33.319818 2923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:33.320797 2923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:33.322242 2923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:33.322348 2923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:33.322499 2923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:33.322578 2923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:33.358647 2923 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39637
I20260620 02:28:33.358696 3050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39637 every 8 connection(s)
I20260620 02:28:33.360049 2923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:33.365298 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 2923
I20260620 02:28:33.365545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:33.366325 388 external_mini_cluster.cc:1428] Setting key 40bf9026bafc334f13e0b45d8433df36
Jun 20 02:28:33 dist-test-slave-r706 krb5kdc[2817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:33.384850 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:33.363070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47709 (local address 127.0.97.62:43017)
0620 02:28:33.363527 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:33.363531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:33.364519 (+ 988us) server_negotiation.cc:407] Connection header received
0620 02:28:33.367218 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:33.367224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:33.367318 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:33.367452 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:33.369413 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.370158 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.372854 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.373378 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.373499 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:33.378375 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:33.378381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:33.378384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:33.378426 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:33.381154 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:33.382247 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:33.382251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:33.382253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:33.382369 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:33.382959 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:33.382963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:33.382965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:33.383229 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:33.383349 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:33.384403 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 02:28:33.384584 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260620 02:28:33.386667 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:33.363663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43017 (local address 127.0.97.1:47709)
0620 02:28:33.364354 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:33.364392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:33.366907 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:33.367710 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:33.367718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:33.368286 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:33.369208 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:33.369240 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.370341 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.370350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:33.372634 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:33.372654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.373629 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.373639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:33.374000 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:33.374808 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:33.374843 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:33.378099 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:33.381320 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:33.381334 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:33.381357 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:33.382111 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:33.382492 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:33.382498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:33.382502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:33.382820 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:33.383371 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:33.383382 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:33.383947 (+ 565us) client_negotiation.cc:769] Sending connection context
0620 02:28:33.384318 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 02:28:33.385032 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":299,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:28:33.388581 3051 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43017
I20260620 02:28:33.388900 3051 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:33.389680 3051 heartbeater.cc:511] Master 127.0.97.62:43017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:28:33.392201 2862 ts_manager.cc:194] Registered new tserver with Master: 63729287727f4b78afcaed4cf89823d4 (127.0.97.1:39637)
I20260620 02:28:33.394450 2862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:33.417399 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43017
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:33.754712 3057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:33.755019 3057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:33.755220 3057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:33.762897 3057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:33.763051 3057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:33.763247 3057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:33.773451 3057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:33.774824 3057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:33.776132 3057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:33.790010 3062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:33.789973 3065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:33.789918 3063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:33.791293 3057 server_base.cc:1034] running on GCE node
I20260620 02:28:33.792053 3057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:33.793164 3057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:33.794449 3057 hybrid_clock.cc:648] HybridClock initialized: now 1781922513794355 us; error 65 us; skew 500 ppm
I20260620 02:28:33.794795 3057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:33 dist-test-slave-r706 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:33.801508 3057 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:33.803121 3057 webserver.cc:460] Webserver started at http://127.0.97.2:33897/ using document root <none> and password file <none>
I20260620 02:28:33.803810 3057 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:33.803920 3057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:33.804188 3057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:33.807262 3057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "d48e1e3b93f249bf822c5ad1c9f256f1"
format_stamp: "Formatted at 2026-06-20 02:28:33 on dist-test-slave-r706"
server_key: "bd3bb60731d70ef5c6d148512c5f2ae1"
server_key_iv: "d6d7cb8eef76ebbd5be585b093e9315f"
server_key_version: "encryptionkey@0"
I20260620 02:28:33.808012 3057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d48e1e3b93f249bf822c5ad1c9f256f1"
format_stamp: "Formatted at 2026-06-20 02:28:33 on dist-test-slave-r706"
server_key: "bd3bb60731d70ef5c6d148512c5f2ae1"
server_key_iv: "d6d7cb8eef76ebbd5be585b093e9315f"
server_key_version: "encryptionkey@0"
I20260620 02:28:33.813480 3057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 02:28:33.816941 3072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:33.818363 3057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:33.818495 3057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "d48e1e3b93f249bf822c5ad1c9f256f1"
format_stamp: "Formatted at 2026-06-20 02:28:33 on dist-test-slave-r706"
server_key: "bd3bb60731d70ef5c6d148512c5f2ae1"
server_key_iv: "d6d7cb8eef76ebbd5be585b093e9315f"
server_key_version: "encryptionkey@0"
I20260620 02:28:33.818660 3057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:33.833814 3057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:33.838181 3057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:33.838456 3057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:33.839466 3057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:33.840750 3057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:33.840823 3057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:33.840940 3057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:33.841071 3057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:33.877144 3057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34777
I20260620 02:28:33.877213 3184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34777 every 8 connection(s)
I20260620 02:28:33.878465 3057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:33.887543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3057
I20260620 02:28:33.887766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:33.888285 388 external_mini_cluster.cc:1428] Setting key 97119c2d1bfd24dfecfb627b067500cb
Jun 20 02:28:33 dist-test-slave-r706 krb5kdc[2817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:33.900166 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:33.881257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55021 (local address 127.0.97.62:43017)
0620 02:28:33.881451 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:33.881455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:33.882772 (+ 1317us) server_negotiation.cc:407] Connection header received
0620 02:28:33.884735 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:33.884741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:33.884835 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:33.884953 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:33.886457 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.887303 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.889468 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.890061 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.890211 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:33.894243 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:33.894248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:33.894252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:33.894285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:33.896957 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:33.897786 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:33.897792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:33.897802 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:33.897933 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:33.898413 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:33.898422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:33.898424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:33.898678 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:33.898848 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:33.899559 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 02:28:33.899793 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":66}
I20260620 02:28:33.901417 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:33.881772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43017 (local address 127.0.97.2:55021)
0620 02:28:33.882603 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:33.882641 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:33.884455 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:33.885203 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:33.885210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:33.885717 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:33.886284 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:33.886300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.887476 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.887482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:33.889311 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:33.889330 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:33.890324 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:33.890332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:33.890582 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:33.891316 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:33.891342 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:33.893975 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:33.897176 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:33.897186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:33.897204 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:33.897651 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:33.898049 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:33.898055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:33.898059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:33.898278 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:33.898814 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:33.898822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:33.899187 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 02:28:33.899458 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 02:28:33.900109 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":361,"threads_started":1}
I20260620 02:28:33.902980 3185 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43017
I20260620 02:28:33.903318 3185 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:33.904076 3185 heartbeater.cc:511] Master 127.0.97.62:43017 requested a full tablet report, sending...
I20260620 02:28:33.905531 2862 ts_manager.cc:194] Registered new tserver with Master: d48e1e3b93f249bf822c5ad1c9f256f1 (127.0.97.2:34777)
I20260620 02:28:33.906342 2862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55021
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:33.932969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43017
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:34.278096 3190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:34.278462 3190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:34.278650 3190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:34.286770 3190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:34.286950 3190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:34.287155 3190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:34.297520 3190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:34.298874 3190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:34.300218 3190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:34.316636 3198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:34.316682 3196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:34.316725 3195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:34.318027 3190 server_base.cc:1034] running on GCE node
I20260620 02:28:34.320986 3190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:34.322352 3190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:34.323676 3190 hybrid_clock.cc:648] HybridClock initialized: now 1781922514323602 us; error 76 us; skew 500 ppm
I20260620 02:28:34.324077 3190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:34 dist-test-slave-r706 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:34.328109 3190 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:34.330507 3190 webserver.cc:460] Webserver started at http://127.0.97.3:43479/ using document root <none> and password file <none>
I20260620 02:28:34.331367 3190 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:34.331561 3190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:34.331964 3190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:34.335393 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "6d4572f7982f4fe1b10b1b52fc546faf"
format_stamp: "Formatted at 2026-06-20 02:28:34 on dist-test-slave-r706"
server_key: "834993450d465d2be84d2954804f96d9"
server_key_iv: "708fe9d247f2361976ad9a6aef670266"
server_key_version: "encryptionkey@0"
I20260620 02:28:34.336282 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6d4572f7982f4fe1b10b1b52fc546faf"
format_stamp: "Formatted at 2026-06-20 02:28:34 on dist-test-slave-r706"
server_key: "834993450d465d2be84d2954804f96d9"
server_key_iv: "708fe9d247f2361976ad9a6aef670266"
server_key_version: "encryptionkey@0"
I20260620 02:28:34.342334 3190 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:28:34.347090 3205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:34.348600 3190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:28:34.348816 3190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "6d4572f7982f4fe1b10b1b52fc546faf"
format_stamp: "Formatted at 2026-06-20 02:28:34 on dist-test-slave-r706"
server_key: "834993450d465d2be84d2954804f96d9"
server_key_iv: "708fe9d247f2361976ad9a6aef670266"
server_key_version: "encryptionkey@0"
I20260620 02:28:34.349112 3190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:34.368167 3190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:34.372817 3190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:34.373275 3190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:34.374341 3190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:34.376358 3190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:34.376467 3190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:34.376633 3190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:34.376706 3190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:34.398361 3051 heartbeater.cc:503] Master 127.0.97.62:43017 was elected leader, sending a full tablet report...
I20260620 02:28:34.418185 3190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41763
I20260620 02:28:34.418290 3317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41763 every 8 connection(s)
I20260620 02:28:34.419538 3190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:34.424964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3190
I20260620 02:28:34.425210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:34.425770 388 external_mini_cluster.cc:1428] Setting key a963b96f276c7701c267037eaa65bcf3
Jun 20 02:28:34 dist-test-slave-r706 krb5kdc[2817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:34.443938 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:34.423086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50955 (local address 127.0.97.62:43017)
0620 02:28:34.423662 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:34.423669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:34.424799 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 02:28:34.427403 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:34.427408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:34.427478 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:34.427567 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:34.429461 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:34.430309 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:34.432031 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:34.432427 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:34.432644 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:34.436889 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:34.436895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:34.436898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:34.436932 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:34.440435 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:34.441586 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:34.441594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:34.441604 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:34.441754 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:34.442353 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:34.442358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:34.442360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:34.442656 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:34.442797 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:34.443468 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 02:28:34.443643 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260620 02:28:34.444931 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:34.423701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43017 (local address 127.0.97.3:50955)
0620 02:28:34.424564 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:34.424642 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:34.427110 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:34.427874 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:34.427883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:34.428480 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:34.429259 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:34.429281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:34.430477 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:34.430481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:34.431823 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:34.431836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:34.432829 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:34.432836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:34.433150 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:34.433963 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:34.433993 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:34.436614 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:34.440628 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:34.440638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:34.440656 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:34.441407 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:34.441921 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:34.441929 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:34.441934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:34.442220 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:34.442758 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:34.442765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:34.443109 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 02:28:34.443372 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 02:28:34.443791 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":340,"threads_started":1}
I20260620 02:28:34.446427 3318 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43017
I20260620 02:28:34.446767 3318 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:34.447537 3318 heartbeater.cc:511] Master 127.0.97.62:43017 requested a full tablet report, sending...
I20260620 02:28:34.449209 2862 ts_manager.cc:194] Registered new tserver with Master: 6d4572f7982f4fe1b10b1b52fc546faf (127.0.97.3:41763)
I20260620 02:28:34.450099 2862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50955
I20260620 02:28:34.455194 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:28:34.491479 3320 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:34.488708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.0.97.62:43017)
0620 02:28:34.488902 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:34.488907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:34.489493 (+ 586us) server_negotiation.cc:407] Connection header received
0620 02:28:34.489725 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:34.489730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:34.489809 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:34.489917 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:34.491256 (+ 1339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48082: BlockingRecv error: recv got EOF from 127.0.0.1:48082 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:28:34.495664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2923
W20260620 02:28:34.499588 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2923: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39637: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:28:34.499814 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 2923
I20260620 02:28:34.516474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3057
W20260620 02:28:34.519210 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3057: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34777: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:28:34.519403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3057
I20260620 02:28:34.535346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3190
W20260620 02:28:34.538887 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3190: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41763: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:28:34.539043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3190
I20260620 02:28:34.555852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 2830
I20260620 02:28:34.557379 2892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:34.669709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 2830
2026-06-20T02:28:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3753 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:35 dist-test-slave-r706 krb5kdc[3336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:35 dist-test-slave-r706 krb5kdc[3336](info): set up 2 sockets
Jun 20 02:28:35 dist-test-slave-r706 krb5kdc[3336](info): commencing operation
krb5kdc: starting...
W20260620 02:28:36.025645 388 mini_kdc.cc:121] Time spent starting KDC: real 1.312s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:36 dist-test-slave-r706 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:36.233935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40031
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:36.568200 3349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:36.568549 3349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:36.568727 3349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:36.576257 3349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:36.576385 3349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:36.576467 3349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:36.576572 3349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:36.576661 3349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:36.587373 3349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:36.589042 3349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:36.590495 3349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:36.602600 3357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:36.602840 3355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:36.603353 3354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:36.603981 3349 server_base.cc:1034] running on GCE node
I20260620 02:28:36.604758 3349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:36.606256 3349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:36.607467 3349 hybrid_clock.cc:648] HybridClock initialized: now 1781922516607429 us; error 38 us; skew 500 ppm
I20260620 02:28:36.607851 3349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:36 dist-test-slave-r706 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:36.611075 3349 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:36.612514 3349 webserver.cc:460] Webserver started at http://127.0.97.62:40113/ using document root <none> and password file <none>
I20260620 02:28:36.613283 3349 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:36.613392 3349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:36.613646 3349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:36.616626 3349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "99206adaea4845759497343f7b8413d8"
format_stamp: "Formatted at 2026-06-20 02:28:36 on dist-test-slave-r706"
server_key: "bc38571df355ae79d92478b41332a1dd"
server_key_iv: "661ff9135ec8c40a5fc9d4ee78645217"
server_key_version: "encryptionkey@0"
I20260620 02:28:36.617501 3349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "99206adaea4845759497343f7b8413d8"
format_stamp: "Formatted at 2026-06-20 02:28:36 on dist-test-slave-r706"
server_key: "bc38571df355ae79d92478b41332a1dd"
server_key_iv: "661ff9135ec8c40a5fc9d4ee78645217"
server_key_version: "encryptionkey@0"
I20260620 02:28:36.623613 3349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260620 02:28:36.627521 3364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:36.629235 3349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:36.629405 3349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "99206adaea4845759497343f7b8413d8"
format_stamp: "Formatted at 2026-06-20 02:28:36 on dist-test-slave-r706"
server_key: "bc38571df355ae79d92478b41332a1dd"
server_key_iv: "661ff9135ec8c40a5fc9d4ee78645217"
server_key_version: "encryptionkey@0"
I20260620 02:28:36.629595 3349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:36.657598 3349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:36.661913 3349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:36.662168 3349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:36.681638 3349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40031
I20260620 02:28:36.681674 3415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40031 every 8 connection(s)
I20260620 02:28:36.683127 3349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:36.685771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3349
I20260620 02:28:36.686008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:36.686847 388 external_mini_cluster.cc:1428] Setting key 96127d37d97f8453f30e529e39188bf7
I20260620 02:28:36.687872 3416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:28:36 dist-test-slave-r706 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:36.702466 3416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8: Bootstrap starting.
I20260620 02:28:36.706866 3416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:36.708216 3416 log.cc:824] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:36.711892 3416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8: No bootstrap required, opened a new log
I20260620 02:28:36.712173 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:36.688677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.97.62:40031)
0620 02:28:36.689573 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:36.689597 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:36.689653 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:28:36.691869 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:36.691900 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:36.692468 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:36.692955 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:36.694401 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:36.696531 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:36.697885 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:36.698952 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:36.699164 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:36.703228 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:36.703240 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:36.703262 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:36.703307 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:36.707481 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:36.708255 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:36.708277 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:36.708285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:36.708499 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:36.709083 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:36.709087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:36.709089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:36.709638 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:36.709958 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:36.710421 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 02:28:36.710863 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":381,"threads_started":1}
I20260620 02:28:36.716125 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:28:36.719990 3416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99206adaea4845759497343f7b8413d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40031 } }
I20260620 02:28:36.720376 3416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:36.720520 3416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99206adaea4845759497343f7b8413d8, State: Initialized, Role: FOLLOWER
I20260620 02:28:36.721416 3416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99206adaea4845759497343f7b8413d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40031 } }
I20260620 02:28:36.721732 3416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:36.721922 3416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:36.722121 3416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:36.725253 3416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99206adaea4845759497343f7b8413d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40031 } }
I20260620 02:28:36.725912 3416 leader_election.cc:302] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99206adaea4845759497343f7b8413d8; no voters:
I20260620 02:28:36.726671 3416 leader_election.cc:288] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:36.726804 3422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:36.728222 3422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [term 1 LEADER]: Becoming Leader. State: Replica: 99206adaea4845759497343f7b8413d8, State: Running, Role: LEADER
I20260620 02:28:36.728868 3422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99206adaea4845759497343f7b8413d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40031 } }
I20260620 02:28:36.729137 3416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:36.731734 3424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99206adaea4845759497343f7b8413d8. Latest consensus state: current_term: 1 leader_uuid: "99206adaea4845759497343f7b8413d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99206adaea4845759497343f7b8413d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40031 } } }
I20260620 02:28:36.732085 3424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:36.732870 3423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99206adaea4845759497343f7b8413d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99206adaea4845759497343f7b8413d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40031 } } }
I20260620 02:28:36.733260 3423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:36.741855 3428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:36.747709 3428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:36.766661 3428 catalog_manager.cc:1347] Generated new cluster ID: 56ad1973d13f42deac6b42e9c2d73a12
I20260620 02:28:36.766961 3428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:36.785116 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:36.775587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.97.62:40031)
0620 02:28:36.775812 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:36.775821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:36.776424 (+ 603us) server_negotiation.cc:407] Connection header received
0620 02:28:36.776607 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:36.776613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:36.776693 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:36.776805 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:36.778209 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:36.779349 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:36.781283 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:36.781895 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:36.782024 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:36.783311 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:36.783318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:36.783322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:36.784214 (+ 892us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:36.784269 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:36.784295 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:36.784395 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:36.784567 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 02:28:36.784669 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":89}
I20260620 02:28:36.798583 3428 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:36.800011 3428 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:36.807006 3428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99206adaea4845759497343f7b8413d8: Generated new TSK 0
I20260620 02:28:36.807994 3428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:28:36.820477 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3349
I20260620 02:28:36.821830 3411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:36.915189 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3349
2026-06-20T02:28:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2239 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:37 dist-test-slave-r706 krb5kdc[3450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:37 dist-test-slave-r706 krb5kdc[3450](info): set up 2 sockets
Jun 20 02:28:37 dist-test-slave-r706 krb5kdc[3450](info): commencing operation
krb5kdc: starting...
W20260620 02:28:37.997573 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:38 dist-test-slave-r706 krb5kdc[3450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:38.210273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40463
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:38.534945 3463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:38.535246 3463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:38.535406 3463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:38.542867 3463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:38.542987 3463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:38.543073 3463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:38.543159 3463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:38.543229 3463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:38.553414 3463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:38.554774 3463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:38.556138 3463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:38.568622 3468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:38.568660 3469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:38.568576 3471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:38.568956 3463 server_base.cc:1034] running on GCE node
I20260620 02:28:38.570086 3463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:38.571569 3463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:38.572831 3463 hybrid_clock.cc:648] HybridClock initialized: now 1781922518572781 us; error 45 us; skew 500 ppm
I20260620 02:28:38.573282 3463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:38 dist-test-slave-r706 krb5kdc[3450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:38.576639 3463 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:38.578263 3463 webserver.cc:460] Webserver started at http://127.0.97.62:37685/ using document root <none> and password file <none>
I20260620 02:28:38.578938 3463 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:38.579041 3463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:38.579308 3463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:38.582322 3463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "c5f7338d8c8d4e72a14a6033ba9e2353"
format_stamp: "Formatted at 2026-06-20 02:28:38 on dist-test-slave-r706"
server_key: "9b688cc87c4a144f6bcf8e27da681077"
server_key_iv: "afb229d59f1f5715de9fa21e8fa2554d"
server_key_version: "encryptionkey@0"
I20260620 02:28:38.583043 3463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5f7338d8c8d4e72a14a6033ba9e2353"
format_stamp: "Formatted at 2026-06-20 02:28:38 on dist-test-slave-r706"
server_key: "9b688cc87c4a144f6bcf8e27da681077"
server_key_iv: "afb229d59f1f5715de9fa21e8fa2554d"
server_key_version: "encryptionkey@0"
I20260620 02:28:38.588169 3463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:28:38.591590 3478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:38.592927 3463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:38.593160 3463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "c5f7338d8c8d4e72a14a6033ba9e2353"
format_stamp: "Formatted at 2026-06-20 02:28:38 on dist-test-slave-r706"
server_key: "9b688cc87c4a144f6bcf8e27da681077"
server_key_iv: "afb229d59f1f5715de9fa21e8fa2554d"
server_key_version: "encryptionkey@0"
I20260620 02:28:38.593361 3463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:38.608346 3463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:38.612602 3463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:38.612914 3463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:38.631677 3463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40463
I20260620 02:28:38.631701 3529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40463 every 8 connection(s)
I20260620 02:28:38.633160 3463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:38.636948 3530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:38.640265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3463
I20260620 02:28:38.640506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:38.641381 388 external_mini_cluster.cc:1428] Setting key b142a6e256603e6541e5a40df0423a5d
I20260620 02:28:38.650076 3530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353: Bootstrap starting.
I20260620 02:28:38.653409 3530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:38.654456 3530 log.cc:824] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:28:38 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:38.657697 3530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353: No bootstrap required, opened a new log
I20260620 02:28:38.664894 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:38.643842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.0.97.62:40463)
0620 02:28:38.644699 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:38.644713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:38.644755 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:28:38.646548 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:38.646573 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:38.647003 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:38.647466 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:38.648965 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:38.650469 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:38.651927 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:38.652598 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:38.652792 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:38.657453 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:38.657462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:38.657477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:38.657521 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:38.660681 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:38.661512 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:38.661520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:38.661526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:38.661684 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:38.662078 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:38.662084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:38.662087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:38.662652 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:38.663006 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:38.663404 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 02:28:38.663851 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":382,"threads_started":1}
I20260620 02:28:38.665886 3530 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40463 } }
I20260620 02:28:38.666262 3530 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:38.666401 3530 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f7338d8c8d4e72a14a6033ba9e2353, State: Initialized, Role: FOLLOWER
I20260620 02:28:38.666945 3530 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40463 } }
I20260620 02:28:38.667199 3530 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:38.667346 3530 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:38.667529 3530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:38.669921 3530 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40463 } }
I20260620 02:28:38.670403 3530 leader_election.cc:302] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5f7338d8c8d4e72a14a6033ba9e2353; no voters:
I20260620 02:28:38.670961 3530 leader_election.cc:288] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:38.671114 3535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:38.671798 3535 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [term 1 LEADER]: Becoming Leader. State: Replica: c5f7338d8c8d4e72a14a6033ba9e2353, State: Running, Role: LEADER
I20260620 02:28:38.672300 3535 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40463 } }
I20260620 02:28:38.673199 3530 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:38.675441 3537 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5f7338d8c8d4e72a14a6033ba9e2353. Latest consensus state: current_term: 1 leader_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40463 } } }
I20260620 02:28:38.675777 3537 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:38.675377 3536 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f7338d8c8d4e72a14a6033ba9e2353" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40463 } } }
I20260620 02:28:38.676191 3536 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:38.677480 3540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:38.682082 3540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:38.691226 3540 catalog_manager.cc:1347] Generated new cluster ID: 56063097ae8d40528835179740a7b4b8
I20260620 02:28:38.691346 3540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:38.712430 3540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:38.714262 3540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:38.725167 3540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5f7338d8c8d4e72a14a6033ba9e2353: Generated new TSK 0
I20260620 02:28:38.726377 3540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:38.789592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40463
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:39.130316 3556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:39.130606 3556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:39.130770 3556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:39.138427 3556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:39.138550 3556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:39.138725 3556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:39.148702 3556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:39.150082 3556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:39.151543 3556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:39.165622 3561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:39.165683 3562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:39.165831 3564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:39.171214 3556 server_base.cc:1034] running on GCE node
I20260620 02:28:39.171890 3556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:39.172936 3556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:39.174300 3556 hybrid_clock.cc:648] HybridClock initialized: now 1781922519174230 us; error 38 us; skew 500 ppm
I20260620 02:28:39.174749 3556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:39 dist-test-slave-r706 krb5kdc[3450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:39.185864 3556 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:39.187659 3556 webserver.cc:460] Webserver started at http://127.0.97.1:38817/ using document root <none> and password file <none>
I20260620 02:28:39.188464 3556 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:39.188622 3556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:39.188941 3556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:39.192101 3556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "22970e8f60b340768a7d1252d2313739"
format_stamp: "Formatted at 2026-06-20 02:28:39 on dist-test-slave-r706"
server_key: "32556061fc56f5329b0916a3fd64827c"
server_key_iv: "5ccaf4b2b8b485908ede9021a97d2f10"
server_key_version: "encryptionkey@0"
I20260620 02:28:39.192943 3556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "22970e8f60b340768a7d1252d2313739"
format_stamp: "Formatted at 2026-06-20 02:28:39 on dist-test-slave-r706"
server_key: "32556061fc56f5329b0916a3fd64827c"
server_key_iv: "5ccaf4b2b8b485908ede9021a97d2f10"
server_key_version: "encryptionkey@0"
I20260620 02:28:39.198480 3556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:28:39.202221 3571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:39.203778 3556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:39.203992 3556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "22970e8f60b340768a7d1252d2313739"
format_stamp: "Formatted at 2026-06-20 02:28:39 on dist-test-slave-r706"
server_key: "32556061fc56f5329b0916a3fd64827c"
server_key_iv: "5ccaf4b2b8b485908ede9021a97d2f10"
server_key_version: "encryptionkey@0"
I20260620 02:28:39.204209 3556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:39.217504 3556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:39.221634 3556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:39.221947 3556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:39.222890 3556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:39.224375 3556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:39.224486 3556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:39.224649 3556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:39.224730 3556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:39.259814 3556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37813
I20260620 02:28:39.259866 3683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37813 every 8 connection(s)
I20260620 02:28:39.261240 3556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:39.261842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3556
I20260620 02:28:39.262074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:39.262955 388 external_mini_cluster.cc:1428] Setting key 187f4a4bd67cdf18b1233c89d74ea856
Jun 20 02:28:39 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:28:39.285992 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:39.264101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46329 (local address 127.0.97.62:40463)
0620 02:28:39.264639 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:39.264644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:39.265975 (+ 1331us) server_negotiation.cc:407] Connection header received
0620 02:28:39.268915 (+ 2940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:39.268919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:39.269066 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:39.269178 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:39.271337 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.272074 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.274490 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.274915 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.275022 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:39.279645 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:39.279650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:39.279654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:39.279688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:39.282624 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:39.283464 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:39.283469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:39.283472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:39.283593 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:39.284065 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:39.284075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:39.284077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:39.284388 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:39.284553 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:39.285354 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 02:28:39.285617 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":328,"threads_started":1}
I20260620 02:28:39.286944 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:39.264780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40463 (local address 127.0.97.1:46329)
0620 02:28:39.265782 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:39.265846 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:39.268610 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:39.269456 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:39.269471 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:39.270316 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:39.271139 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:39.271165 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.272253 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.272261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:39.274261 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:39.274280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.275168 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.275178 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:39.275545 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:39.276294 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:39.276329 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:39.279360 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:39.282804 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:39.282816 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:39.282838 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:39.283309 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:39.283714 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:39.283720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:39.283724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:39.283929 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:39.284536 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:39.284545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:39.284900 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 02:28:39.285240 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 02:28:39.285813 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":420,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:28:39.288664 3684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40463
I20260620 02:28:39.289060 3684 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:39.289857 3684 heartbeater.cc:511] Master 127.0.97.62:40463 requested a full tablet report, sending...
I20260620 02:28:39.292013 3495 ts_manager.cc:194] Registered new tserver with Master: 22970e8f60b340768a7d1252d2313739 (127.0.97.1:37813)
I20260620 02:28:39.293582 3495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:39.313544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40463
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:39.636300 3690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:39.636603 3690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:39.636768 3690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:39.644218 3690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:39.644345 3690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:39.644548 3690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:39.654929 3690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:39.656289 3690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:39.657703 3690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:39.671082 3695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:39.671170 3698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:39.671136 3696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:39.672041 3690 server_base.cc:1034] running on GCE node
I20260620 02:28:39.672622 3690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:39.673714 3690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:39.674906 3690 hybrid_clock.cc:648] HybridClock initialized: now 1781922519674850 us; error 67 us; skew 500 ppm
I20260620 02:28:39.675195 3690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:39 dist-test-slave-r706 krb5kdc[3450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:39.678309 3690 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:39.679844 3690 webserver.cc:460] Webserver started at http://127.0.97.2:33247/ using document root <none> and password file <none>
I20260620 02:28:39.680565 3690 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:39.680711 3690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:39.681083 3690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:39.684028 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "9247ad82ed76483681445100f6d9866d"
format_stamp: "Formatted at 2026-06-20 02:28:39 on dist-test-slave-r706"
server_key: "5db4899651972f58000f02cf7628f2cf"
server_key_iv: "ff632de11ac8e741a509b77027ce83f6"
server_key_version: "encryptionkey@0"
I20260620 02:28:39.684827 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9247ad82ed76483681445100f6d9866d"
format_stamp: "Formatted at 2026-06-20 02:28:39 on dist-test-slave-r706"
server_key: "5db4899651972f58000f02cf7628f2cf"
server_key_iv: "ff632de11ac8e741a509b77027ce83f6"
server_key_version: "encryptionkey@0"
I20260620 02:28:39.690155 3690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:28:39.694162 3705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:39.695449 3690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:39.695619 3690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "9247ad82ed76483681445100f6d9866d"
format_stamp: "Formatted at 2026-06-20 02:28:39 on dist-test-slave-r706"
server_key: "5db4899651972f58000f02cf7628f2cf"
server_key_iv: "ff632de11ac8e741a509b77027ce83f6"
server_key_version: "encryptionkey@0"
I20260620 02:28:39.695818 3690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:39.714522 3690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:39.718736 3690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:39.719072 3690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:39.720134 3690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:39.721904 3690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:39.721982 3690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:39.722072 3690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:39.722164 3690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:39.756223 3690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33625
I20260620 02:28:39.756273 3817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33625 every 8 connection(s)
I20260620 02:28:39.757637 3690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:39.763748 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3690
I20260620 02:28:39.763994 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:39.764590 388 external_mini_cluster.cc:1428] Setting key 779ea3bc7bbd05722a2528e55c02d8e5
Jun 20 02:28:39 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:39.780192 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:39.760456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35043 (local address 127.0.97.62:40463)
0620 02:28:39.760647 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:39.760652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:39.761838 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 02:28:39.763807 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:39.763812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:39.763922 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:39.764051 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:39.765868 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.766730 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.768329 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.769129 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.769276 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:39.773026 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:39.773038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:39.773048 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:39.773101 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:39.776423 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:39.777490 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:39.777495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:39.777497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:39.777637 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:39.778245 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:39.778249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:39.778251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:39.778532 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:39.778649 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:39.779687 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0620 02:28:39.779935 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:28:39.781759 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:39.760902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40463 (local address 127.0.97.2:35043)
0620 02:28:39.761666 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:39.761713 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:39.763523 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:39.764297 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:39.764305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:39.764891 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:39.765672 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:39.765715 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.766884 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.766889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:39.768169 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:39.768181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:39.769341 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:39.769347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:39.769559 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:39.770363 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:39.770391 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:39.772707 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:39.776593 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:39.776606 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:39.776628 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:39.777351 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:39.777772 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:39.777780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:39.777786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:39.778055 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:39.778662 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:39.778674 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:39.779211 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 02:28:39.779531 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 02:28:39.780202 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":281,"threads_started":1}
I20260620 02:28:39.783377 3818 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40463
I20260620 02:28:39.783716 3818 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:39.784677 3818 heartbeater.cc:511] Master 127.0.97.62:40463 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:28:39.786363 3495 ts_manager.cc:194] Registered new tserver with Master: 9247ad82ed76483681445100f6d9866d (127.0.97.2:33625)
I20260620 02:28:39.787271 3495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:39.813494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40463
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:40.148627 3823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:40.148937 3823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:40.149129 3823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:40.156926 3823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:40.157140 3823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:40.157267 3823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:40.167419 3823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:40.168731 3823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:40.170149 3823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:40.184372 3828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:40.184321 3829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:40.184705 3831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:40.185715 3823 server_base.cc:1034] running on GCE node
I20260620 02:28:40.186407 3823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:40.187577 3823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:40.188920 3823 hybrid_clock.cc:648] HybridClock initialized: now 1781922520188845 us; error 64 us; skew 500 ppm
I20260620 02:28:40.189342 3823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:40.192951 3823 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:40.195058 3823 webserver.cc:460] Webserver started at http://127.0.97.3:43455/ using document root <none> and password file <none>
I20260620 02:28:40.195847 3823 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:40.196019 3823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:40.196382 3823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:40.200088 3823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "d26058e42a2543f5ab0aa81b07402647"
format_stamp: "Formatted at 2026-06-20 02:28:40 on dist-test-slave-r706"
server_key: "83c25a098fee6fda1f922bf3a957a7a2"
server_key_iv: "b8c0aba62f6212bb2506f95303282d4d"
server_key_version: "encryptionkey@0"
I20260620 02:28:40.201166 3823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d26058e42a2543f5ab0aa81b07402647"
format_stamp: "Formatted at 2026-06-20 02:28:40 on dist-test-slave-r706"
server_key: "83c25a098fee6fda1f922bf3a957a7a2"
server_key_iv: "b8c0aba62f6212bb2506f95303282d4d"
server_key_version: "encryptionkey@0"
I20260620 02:28:40.207751 3823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 02:28:40.211810 3838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:40.213691 3823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:40.213920 3823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "d26058e42a2543f5ab0aa81b07402647"
format_stamp: "Formatted at 2026-06-20 02:28:40 on dist-test-slave-r706"
server_key: "83c25a098fee6fda1f922bf3a957a7a2"
server_key_iv: "b8c0aba62f6212bb2506f95303282d4d"
server_key_version: "encryptionkey@0"
I20260620 02:28:40.214165 3823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:40.232232 3823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:40.237216 3823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:40.237594 3823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:40.238677 3823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:40.240324 3823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:40.240458 3823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:40.240633 3823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:40.240737 3823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:40.277215 3823 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33251
I20260620 02:28:40.277262 3950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33251 every 8 connection(s)
I20260620 02:28:40.278635 3823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:40.286063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3823
I20260620 02:28:40.286243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:40.286738 388 external_mini_cluster.cc:1428] Setting key a9e87023a5c445f035b801d9837d8d88
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:40.296655 3684 heartbeater.cc:503] Master 127.0.97.62:40463 was elected leader, sending a full tablet report...
I20260620 02:28:40.301486 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:40.281618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35809 (local address 127.0.97.62:40463)
0620 02:28:40.282122 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:40.282127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:40.283013 (+ 886us) server_negotiation.cc:407] Connection header received
0620 02:28:40.284981 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:40.284987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:40.285141 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:40.285276 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:40.287016 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.288316 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.290432 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.290918 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.291083 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:40.294757 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:40.294763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:40.294766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:40.294800 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:40.298013 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.298817 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.298821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.298824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.298962 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.299352 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.299356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.299358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.299724 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:40.299858 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:40.300738 (+ 880us) server_negotiation.cc:299] Negotiation successful
0620 02:28:40.300985 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":284,"threads_started":1}
I20260620 02:28:40.302505 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:40.282049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40463 (local address 127.0.97.3:35809)
0620 02:28:40.282841 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:40.282890 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:40.284695 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:40.285534 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:40.285544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:40.286075 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:40.286817 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:40.286848 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.288503 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.288508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:40.290252 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:40.290265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.291163 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.291173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:40.291461 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:40.292095 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:40.292119 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:40.294512 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:40.298155 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:40.298180 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:40.298200 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:40.298704 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:40.299064 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:40.299067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:40.299069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:40.299255 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:40.299854 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:40.299860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:40.300248 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 02:28:40.300566 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 02:28:40.301210 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
I20260620 02:28:40.303844 3951 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40463
I20260620 02:28:40.304162 3951 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:40.304949 3951 heartbeater.cc:511] Master 127.0.97.62:40463 requested a full tablet report, sending...
I20260620 02:28:40.306387 3495 ts_manager.cc:194] Registered new tserver with Master: d26058e42a2543f5ab0aa81b07402647 (127.0.97.3:33251)
I20260620 02:28:40.307396 3495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35809
I20260620 02:28:40.314760 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:40.365387 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:40.351068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.0.97.62:40463)
0620 02:28:40.351319 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:40.351326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:40.351630 (+ 304us) server_negotiation.cc:407] Connection header received
0620 02:28:40.351777 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:40.351782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:40.351860 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:40.351952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:40.353390 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.354120 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.355480 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.355958 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.356071 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:40.360183 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:40.360188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:40.360192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:40.360226 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:40.362862 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.363571 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.363575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.363577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.363705 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.364107 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.364110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.364112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.364439 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:40.364616 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:40.364913 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 02:28:40.365168 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 02:28:40.366036 3495 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40850
I20260620 02:28:40.367789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3556
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:40.385742 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:40.369196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.0.97.1:37813)
0620 02:28:40.369899 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:40.369906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:40.369933 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:28:40.369992 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:40.369999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:40.370212 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:40.370347 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:40.371787 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.372869 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.374506 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.375123 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.375226 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:40.380151 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:40.380163 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:40.380176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:40.380215 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:40.383093 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.383820 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.383828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.383830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.383968 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.384336 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.384340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.384342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.384694 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:40.384932 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:40.385223 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 02:28:40.385456 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":366,"threads_started":1}
W20260620 02:28:40.387056 3679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59802
W20260620 02:28:40.388193 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:40.388388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3556
I20260620 02:28:40.404690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3690
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:40.420387 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:40.405727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.0.97.2:33625)
0620 02:28:40.406256 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:40.406263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:40.406288 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:28:40.406351 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:40.406359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:40.406565 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:40.406698 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:40.407864 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.408903 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.410146 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.410680 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.410789 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:40.414801 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:40.414812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:40.414826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:40.414864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:40.417767 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.418528 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.418535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.418538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.418671 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.419033 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.419037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.419039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.419372 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:40.419620 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:40.419889 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 02:28:40.420097 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":259,"threads_started":1}
W20260620 02:28:40.421820 3813 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33876
W20260620 02:28:40.422933 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3690: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:40.423069 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3690
I20260620 02:28:40.440156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3823
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922520, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:40.456861 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:40.441232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.0.97.3:33251)
0620 02:28:40.441811 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:40.441818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:40.441848 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:28:40.441910 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:40.441916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:40.442145 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:40.442275 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:40.443596 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.444628 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.445999 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:40.446580 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:40.446688 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:40.451183 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:40.451197 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:40.451212 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:40.451250 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:40.454344 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.455101 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.455109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.455111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.455240 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:40.455562 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:40.455566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:40.455568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:40.455907 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:40.456137 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:40.456347 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 02:28:40.456573 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
W20260620 02:28:40.458151 3946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50356
W20260620 02:28:40.459333 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3823: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:28:40.459570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3823
I20260620 02:28:40.475632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3463
I20260620 02:28:40.476784 3525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:40.581071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3463
2026-06-20T02:28:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3672 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3969](info): set up 2 sockets
Jun 20 02:28:40 dist-test-slave-r706 krb5kdc[3969](info): commencing operation
krb5kdc: starting...
W20260620 02:28:41.689601 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:41 dist-test-slave-r706 krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:41.912882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34917
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:42.254647 3982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:42.254966 3982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:42.255059 3982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:42.262604 3982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:42.262733 3982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:42.262781 3982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:42.262832 3982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:42.262871 3982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:42.273506 3982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:42.274847 3982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:42.276108 3982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:42.288045 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:42.288166 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:42.288177 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:42.289095 3982 server_base.cc:1034] running on GCE node
I20260620 02:28:42.289963 3982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:42.291496 3982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:42.292784 3982 hybrid_clock.cc:648] HybridClock initialized: now 1781922522292740 us; error 64 us; skew 500 ppm
I20260620 02:28:42.293331 3982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:42 dist-test-slave-r706 krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:42.297219 3982 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:42.298972 3982 webserver.cc:460] Webserver started at http://127.0.97.62:39199/ using document root <none> and password file <none>
I20260620 02:28:42.299713 3982 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:42.299837 3982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:42.300164 3982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:42.303700 3982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "1c3a944f5b5a435e915061be0d33755d"
format_stamp: "Formatted at 2026-06-20 02:28:42 on dist-test-slave-r706"
server_key: "d65bf275df6f44375822a2d497d414da"
server_key_iv: "72ff11fe922a6603ecafb56ad7640e3c"
server_key_version: "encryptionkey@0"
I20260620 02:28:42.304581 3982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c3a944f5b5a435e915061be0d33755d"
format_stamp: "Formatted at 2026-06-20 02:28:42 on dist-test-slave-r706"
server_key: "d65bf275df6f44375822a2d497d414da"
server_key_iv: "72ff11fe922a6603ecafb56ad7640e3c"
server_key_version: "encryptionkey@0"
I20260620 02:28:42.310637 3982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260620 02:28:42.314630 3997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:42.316039 3982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:42.316195 3982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "1c3a944f5b5a435e915061be0d33755d"
format_stamp: "Formatted at 2026-06-20 02:28:42 on dist-test-slave-r706"
server_key: "d65bf275df6f44375822a2d497d414da"
server_key_iv: "72ff11fe922a6603ecafb56ad7640e3c"
server_key_version: "encryptionkey@0"
I20260620 02:28:42.316388 3982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:42.331357 3982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:42.335858 3982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:42.336105 3982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:42.355688 3982 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34917
I20260620 02:28:42.355727 4048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34917 every 8 connection(s)
I20260620 02:28:42.357182 3982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:42.361166 4049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:42.364087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 3982
I20260620 02:28:42.364382 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:42.365149 388 external_mini_cluster.cc:1428] Setting key fc71d85ff5456e1d720888febdfe3ef0
I20260620 02:28:42.375082 4049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d: Bootstrap starting.
I20260620 02:28:42.379483 4049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:42.381042 4049 log.cc:824] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:28:42 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:42.384268 4049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d: No bootstrap required, opened a new log
I20260620 02:28:42.390995 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:42.367383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.97.62:34917)
0620 02:28:42.368245 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:42.368257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:42.368303 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:28:42.370071 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:42.370094 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:42.370504 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:42.370852 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:42.373088 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:42.374439 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:42.376328 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:42.377095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:42.377236 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:42.382518 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:42.382533 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:42.382559 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:42.382604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:42.386095 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:42.387103 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:42.387111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:42.387118 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:42.387353 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:42.387847 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:42.387851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:42.387853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:42.388507 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:42.388755 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:42.389367 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 02:28:42.389873 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":406,"threads_started":1}
I20260620 02:28:42.393599 4049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3a944f5b5a435e915061be0d33755d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34917 } }
I20260620 02:28:42.393996 4049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:42.394138 4049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3a944f5b5a435e915061be0d33755d, State: Initialized, Role: FOLLOWER
I20260620 02:28:42.394729 4049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3a944f5b5a435e915061be0d33755d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34917 } }
I20260620 02:28:42.394985 4049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:42.395135 4049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:42.395313 4049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:42.397585 4049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3a944f5b5a435e915061be0d33755d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34917 } }
I20260620 02:28:42.398097 4049 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3a944f5b5a435e915061be0d33755d; no voters:
I20260620 02:28:42.398777 4049 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:42.398859 4054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:42.399605 4054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [term 1 LEADER]: Becoming Leader. State: Replica: 1c3a944f5b5a435e915061be0d33755d, State: Running, Role: LEADER
I20260620 02:28:42.400285 4054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3a944f5b5a435e915061be0d33755d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34917 } }
I20260620 02:28:42.401118 4049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:42.403568 4056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3a944f5b5a435e915061be0d33755d. Latest consensus state: current_term: 1 leader_uuid: "1c3a944f5b5a435e915061be0d33755d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3a944f5b5a435e915061be0d33755d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34917 } } }
I20260620 02:28:42.404080 4056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:42.404575 4055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3a944f5b5a435e915061be0d33755d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3a944f5b5a435e915061be0d33755d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34917 } } }
I20260620 02:28:42.405030 4055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:42.409979 4062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:42.415531 4062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:42.425649 4062 catalog_manager.cc:1347] Generated new cluster ID: ad672be031254bd0944a0e9217ecb22c
I20260620 02:28:42.425807 4062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:42.448537 4062 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:42.450033 4062 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:42.456660 4062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3a944f5b5a435e915061be0d33755d: Generated new TSK 0
I20260620 02:28:42.457711 4062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:42.528836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34917
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:42.890125 4075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:42.890534 4075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:42.890643 4075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:42.898216 4075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:42.898350 4075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:42.898469 4075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:42.908854 4075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:42.910295 4075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:42.911705 4075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:42.925370 4080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:42.925402 4083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:42.925325 4081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:42.926265 4075 server_base.cc:1034] running on GCE node
I20260620 02:28:42.926862 4075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:42.927757 4075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:42.929112 4075 hybrid_clock.cc:648] HybridClock initialized: now 1781922522928932 us; error 57 us; skew 500 ppm
I20260620 02:28:42.929490 4075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:42 dist-test-slave-r706 krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:42.933102 4075 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:42.934803 4075 webserver.cc:460] Webserver started at http://127.0.97.1:42535/ using document root <none> and password file <none>
I20260620 02:28:42.935566 4075 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:42.935714 4075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:42.936097 4075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:42.939460 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "48988a4b2d804a0aad9d45b8f52f9f92"
format_stamp: "Formatted at 2026-06-20 02:28:42 on dist-test-slave-r706"
server_key: "3311cbdea9644dd9123d62f7c9f6d23a"
server_key_iv: "e7f54be6261bf38519057db7d901fd00"
server_key_version: "encryptionkey@0"
I20260620 02:28:42.940462 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "48988a4b2d804a0aad9d45b8f52f9f92"
format_stamp: "Formatted at 2026-06-20 02:28:42 on dist-test-slave-r706"
server_key: "3311cbdea9644dd9123d62f7c9f6d23a"
server_key_iv: "e7f54be6261bf38519057db7d901fd00"
server_key_version: "encryptionkey@0"
I20260620 02:28:42.946748 4075 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260620 02:28:42.950832 4090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:42.952382 4075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:28:42.952585 4075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "48988a4b2d804a0aad9d45b8f52f9f92"
format_stamp: "Formatted at 2026-06-20 02:28:42 on dist-test-slave-r706"
server_key: "3311cbdea9644dd9123d62f7c9f6d23a"
server_key_iv: "e7f54be6261bf38519057db7d901fd00"
server_key_version: "encryptionkey@0"
I20260620 02:28:42.952864 4075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:42.975703 4075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:42.980055 4075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:42.980386 4075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:42.981544 4075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:42.982965 4075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:42.983043 4075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:42.983234 4075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:42.983289 4075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:43.017948 4075 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38347
I20260620 02:28:43.018111 4202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38347 every 8 connection(s)
I20260620 02:28:43.019289 4075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:43.023293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4075
I20260620 02:28:43.023537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:43.024319 388 external_mini_cluster.cc:1428] Setting key 193be1f4834e67f3381748dde3dcf810
Jun 20 02:28:43 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:43.042171 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:43.022150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42679 (local address 127.0.97.62:34917)
0620 02:28:43.022694 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:43.022701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:43.023833 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 02:28:43.026032 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:43.026038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:43.026126 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:43.026260 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:43.027886 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.028632 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.030339 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.030763 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.030907 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:43.035245 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:43.035251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:43.035258 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:43.035295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:43.038068 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:43.039206 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:43.039211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:43.039214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:43.039379 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:43.040017 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:43.040023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:43.040027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:43.040384 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:43.040520 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:43.041650 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0620 02:28:43.041867 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260620 02:28:43.043795 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:43.022706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34917 (local address 127.0.97.1:42679)
0620 02:28:43.023622 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:43.023679 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:43.025738 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:43.026535 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:43.026544 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:43.027103 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:43.027688 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:43.027707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.028805 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.028815 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:43.030171 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:43.030185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.031030 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.031038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:43.031283 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:43.032063 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:43.032091 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:43.034962 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:43.038244 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:43.038257 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:43.038281 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:43.039005 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:43.039519 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:43.039527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:43.039534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:43.039825 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:43.040541 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:43.040553 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:43.041147 (+ 594us) client_negotiation.cc:769] Sending connection context
0620 02:28:43.041522 (+ 375us) client_negotiation.cc:240] Negotiation successful
0620 02:28:43.042187 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":408,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:28:43.045861 4203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34917
I20260620 02:28:43.046233 4203 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:43.047049 4203 heartbeater.cc:511] Master 127.0.97.62:34917 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:28:43.049976 4014 ts_manager.cc:194] Registered new tserver with Master: 48988a4b2d804a0aad9d45b8f52f9f92 (127.0.97.1:38347)
I20260620 02:28:43.052246 4014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:43.074982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34917
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:43.422914 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:43.423225 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:43.423385 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:43.431034 4209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:43.431154 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:43.431334 4209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:43.441464 4209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:43.442873 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:43.444218 4209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:43.459240 4214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:43.459180 4215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:43.459141 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:43.460400 4209 server_base.cc:1034] running on GCE node
I20260620 02:28:43.461186 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:43.462342 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:43.463589 4209 hybrid_clock.cc:648] HybridClock initialized: now 1781922523463527 us; error 57 us; skew 500 ppm
I20260620 02:28:43.463954 4209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:43 dist-test-slave-r706 krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:43.467420 4209 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:43.469125 4209 webserver.cc:460] Webserver started at http://127.0.97.2:42425/ using document root <none> and password file <none>
I20260620 02:28:43.469812 4209 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:43.469918 4209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:43.470192 4209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:43.473237 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "4ceeb67b7d5c49459ff788a869aa5102"
format_stamp: "Formatted at 2026-06-20 02:28:43 on dist-test-slave-r706"
server_key: "22a5011a1665460c73783a41435dad34"
server_key_iv: "5c6925269fb46fcbfe8de1eac8161b1f"
server_key_version: "encryptionkey@0"
I20260620 02:28:43.474028 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4ceeb67b7d5c49459ff788a869aa5102"
format_stamp: "Formatted at 2026-06-20 02:28:43 on dist-test-slave-r706"
server_key: "22a5011a1665460c73783a41435dad34"
server_key_iv: "5c6925269fb46fcbfe8de1eac8161b1f"
server_key_version: "encryptionkey@0"
I20260620 02:28:43.479553 4209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:43.483309 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:43.484794 4209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:43.485067 4209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "4ceeb67b7d5c49459ff788a869aa5102"
format_stamp: "Formatted at 2026-06-20 02:28:43 on dist-test-slave-r706"
server_key: "22a5011a1665460c73783a41435dad34"
server_key_iv: "5c6925269fb46fcbfe8de1eac8161b1f"
server_key_version: "encryptionkey@0"
I20260620 02:28:43.485270 4209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:43.503468 4209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:43.508255 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:43.508580 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:43.509836 4209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:43.511430 4209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:43.511504 4209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:43.511659 4209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:43.511722 4209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:43.548586 4209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46733
I20260620 02:28:43.548684 4336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46733 every 8 connection(s)
I20260620 02:28:43.549979 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:43.557845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4209
I20260620 02:28:43.558068 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:43.558677 388 external_mini_cluster.cc:1428] Setting key 088f2b303c4f6c265952106b6977871e
Jun 20 02:28:43 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:43.573838 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:43.552692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58259 (local address 127.0.97.62:34917)
0620 02:28:43.553199 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:43.553204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:43.554214 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 02:28:43.556046 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:43.556052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:43.556146 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:43.556282 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:43.558265 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.559332 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.561163 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.561893 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.562048 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:43.566749 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:43.566763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:43.566768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:43.566815 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:43.569717 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:43.570994 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:43.570998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:43.571000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:43.571151 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:43.571783 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:43.571787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:43.571790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:43.572104 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:43.572248 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:43.573354 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0620 02:28:43.573545 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":207,"threads_started":1}
I20260620 02:28:43.576004 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:43.553301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34917 (local address 127.0.97.2:58259)
0620 02:28:43.554032 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:43.554071 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:43.555767 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:43.556535 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:43.556545 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:43.557190 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:43.558023 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:43.558055 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.559507 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.559514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:43.560966 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:43.560979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:43.562155 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:43.562164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:43.562457 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:43.563188 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:43.563219 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:43.566281 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:43.569926 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:43.569941 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:43.569964 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:43.570803 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:43.571283 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:43.571292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:43.571299 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:43.571613 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:43.572272 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:43.572285 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:43.572854 (+ 569us) client_negotiation.cc:769] Sending connection context
0620 02:28:43.573255 (+ 401us) client_negotiation.cc:240] Negotiation successful
0620 02:28:43.573950 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260620 02:28:43.577899 4337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34917
I20260620 02:28:43.578305 4337 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:43.579499 4337 heartbeater.cc:511] Master 127.0.97.62:34917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:28:43.581754 4014 ts_manager.cc:194] Registered new tserver with Master: 4ceeb67b7d5c49459ff788a869aa5102 (127.0.97.2:46733)
I20260620 02:28:43.582718 4014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:43.613336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34917
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:43.954735 4343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:43.955107 4343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:43.955312 4343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:43.963115 4343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:43.963357 4343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:43.963526 4343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:43.974443 4343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:43.975782 4343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:43.977192 4343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:43.991951 4348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:43.991864 4351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:43.992215 4349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:43.993625 4343 server_base.cc:1034] running on GCE node
I20260620 02:28:43.994333 4343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:43.995415 4343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:43.996680 4343 hybrid_clock.cc:648] HybridClock initialized: now 1781922523996635 us; error 46 us; skew 500 ppm
I20260620 02:28:43.997131 4343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:43 dist-test-slave-r706 krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:44.000516 4343 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:44.002329 4343 webserver.cc:460] Webserver started at http://127.0.97.3:45495/ using document root <none> and password file <none>
I20260620 02:28:44.003086 4343 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:44.003253 4343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:44.003664 4343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:44.006866 4343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "8c15a883fa08490c8065fe903ef77eeb"
format_stamp: "Formatted at 2026-06-20 02:28:43 on dist-test-slave-r706"
server_key: "76e1dfcb713ae337e16ac960f6777c10"
server_key_iv: "959929898b715453ad57ce7befce377c"
server_key_version: "encryptionkey@0"
I20260620 02:28:44.007735 4343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8c15a883fa08490c8065fe903ef77eeb"
format_stamp: "Formatted at 2026-06-20 02:28:43 on dist-test-slave-r706"
server_key: "76e1dfcb713ae337e16ac960f6777c10"
server_key_iv: "959929898b715453ad57ce7befce377c"
server_key_version: "encryptionkey@0"
I20260620 02:28:44.014089 4343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:28:44.018211 4358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:44.019974 4343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:28:44.020164 4343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "8c15a883fa08490c8065fe903ef77eeb"
format_stamp: "Formatted at 2026-06-20 02:28:43 on dist-test-slave-r706"
server_key: "76e1dfcb713ae337e16ac960f6777c10"
server_key_iv: "959929898b715453ad57ce7befce377c"
server_key_version: "encryptionkey@0"
I20260620 02:28:44.020512 4343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:44.039919 4343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:44.044509 4343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:44.044793 4343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:44.045810 4343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:44.047200 4343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:44.047276 4343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:44.047367 4343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:44.047412 4343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:44.055961 4203 heartbeater.cc:503] Master 127.0.97.62:34917 was elected leader, sending a full tablet report...
I20260620 02:28:44.083921 4343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41877
I20260620 02:28:44.084015 4470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41877 every 8 connection(s)
I20260620 02:28:44.085690 4343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:44.095721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4343
I20260620 02:28:44.095927 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:44.096365 388 external_mini_cluster.cc:1428] Setting key 5ccbf5e15b10c91dcb40e34adc5d563a
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:44.108654 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.089120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53115 (local address 127.0.97.62:34917)
0620 02:28:44.089594 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:44.089599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:44.090765 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 02:28:44.092671 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:44.092676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:44.092783 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:44.092934 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:44.094546 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.095438 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.097411 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.098048 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.098189 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.102403 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:44.102408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:44.102411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:44.102446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:44.105324 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.106142 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.106149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.106158 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.106279 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.106771 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.106775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.106777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.107073 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:44.107240 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:44.108011 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 02:28:44.108264 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260620 02:28:44.109725 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.089690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34917 (local address 127.0.97.3:53115)
0620 02:28:44.090586 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:44.090635 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:44.092377 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:44.093206 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:44.093215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:44.093769 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:44.094365 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:44.094382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.095633 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.095639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:44.097261 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:44.097274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.098336 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.098344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:44.098662 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.099534 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:44.099570 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:44.102149 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:44.105503 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:44.105513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:44.105545 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:44.105997 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:44.106396 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:44.106401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:44.106404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:44.106620 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:44.107212 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:44.107221 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:44.107597 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 02:28:44.107897 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 02:28:44.108473 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":323,"threads_started":1}
I20260620 02:28:44.111303 4471 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34917
I20260620 02:28:44.111649 4471 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:44.112419 4471 heartbeater.cc:511] Master 127.0.97.62:34917 requested a full tablet report, sending...
I20260620 02:28:44.113966 4014 ts_manager.cc:194] Registered new tserver with Master: 8c15a883fa08490c8065fe903ef77eeb (127.0.97.3:41877)
I20260620 02:28:44.114853 4014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53115
I20260620 02:28:44.124964 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[3969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:44.178201 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.161563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.0.97.1:38347)
0620 02:28:44.162296 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:44.162304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:44.162332 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:28:44.162398 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:44.162405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:44.162640 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:44.162834 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:44.164653 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.165794 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.167170 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.167808 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.167976 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.172796 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:44.172810 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:44.172826 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:44.172866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:44.175720 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.176348 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.176355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.176357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.176495 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.176819 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.176822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.176824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.177249 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:44.177491 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:44.177717 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 02:28:44.177910 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":336,"threads_started":1}
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:44.199332 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.182577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.0.97.62:34917)
0620 02:28:44.182798 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:44.182805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:44.182940 (+ 135us) server_negotiation.cc:407] Connection header received
0620 02:28:44.183107 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:44.183111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:44.183177 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:44.183256 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:44.184477 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.185314 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.186623 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.187017 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.187123 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.194117 (+ 6994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:44.194124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:44.194127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:44.194170 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:44.196795 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.197502 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.197506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.197509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.197647 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.198028 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.198031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.198033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.198378 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:44.198585 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:44.198903 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:28:44.199094 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 02:28:44.199945 4014 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57264
I20260620 02:28:44.201097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4075
I20260620 02:28:44.215716 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.202196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.0.97.1:38347)
0620 02:28:44.202433 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:44.202440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:44.202674 (+ 234us) server_negotiation.cc:407] Connection header received
0620 02:28:44.202885 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:44.202890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:44.202965 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:44.203073 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:44.204362 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.205162 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.206427 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.206872 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.207011 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.209801 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:44.209807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:44.209811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:44.209847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:44.212901 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.213740 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.213749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.213753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.213897 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.214279 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.214284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.214288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.214562 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:44.214831 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:44.215141 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 02:28:44.215384 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:28:44.216409 4198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:44.300848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4075
I20260620 02:28:44.317725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4209
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:44.334647 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.318693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.0.97.2:46733)
0620 02:28:44.319290 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:44.319297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:44.319324 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:28:44.319394 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:44.319401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:44.319618 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:44.319832 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:44.321175 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.322223 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.323550 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.324148 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.324255 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.328747 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:44.328760 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:44.328775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:44.328816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:44.331883 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.332613 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.332621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.332623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.332750 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.333253 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.333257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.333259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.333610 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:44.333838 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:44.334155 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:28:44.334360 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":328,"threads_started":1}
I20260620 02:28:44.335994 4332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:44.418807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4209
I20260620 02:28:44.435832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4343
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[3969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:44.453177 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:44.436946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.0.97.3:41877)
0620 02:28:44.437658 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:44.437666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:44.437695 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:28:44.437761 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:44.437768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:44.437981 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:44.438186 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:44.439542 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.440572 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.442167 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:44.442763 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:44.442912 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:44.447551 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:44.447566 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:44.447581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:44.447622 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:44.450419 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.451102 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.451110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.451112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.451233 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:44.451655 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:44.451659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:44.451661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:44.452015 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:44.452274 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:44.452585 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 02:28:44.452795 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":394,"threads_started":1}
I20260620 02:28:44.454484 4466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:44.537706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4343
I20260620 02:28:44.554186 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 3982
I20260620 02:28:44.555469 4044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:44.659678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 3982
2026-06-20T02:28:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4078 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[4491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[4491](info): set up 2 sockets
Jun 20 02:28:44 dist-test-slave-r706 krb5kdc[4491](info): commencing operation
krb5kdc: starting...
W20260620 02:28:45.773756 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:45 dist-test-slave-r706 krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:45.998113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45947
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:46.330686 4504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:46.330999 4504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:46.331178 4504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:46.338534 4504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:46.338654 4504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:46.338749 4504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:46.338845 4504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:46.338943 4504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:46.349253 4504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:46.350677 4504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:46.352123 4504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:46.364764 4509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:46.364795 4512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:46.364857 4510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:46.365767 4504 server_base.cc:1034] running on GCE node
I20260620 02:28:46.366659 4504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:46.368260 4504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:46.369604 4504 hybrid_clock.cc:648] HybridClock initialized: now 1781922526369537 us; error 58 us; skew 500 ppm
I20260620 02:28:46.370038 4504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:46 dist-test-slave-r706 krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:46.374010 4504 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:46.375244 4504 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:46.375344 4504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:46.375609 4504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:46.378873 4504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "4e91dcfc975a4c64a93332b8e31b44b2"
format_stamp: "Formatted at 2026-06-20 02:28:46 on dist-test-slave-r706"
server_key: "8a6c9012262ffc2998beba0c0741a9ba"
server_key_iv: "ce432737d76a59dbbfd7837181033058"
server_key_version: "encryptionkey@0"
I20260620 02:28:46.379666 4504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "4e91dcfc975a4c64a93332b8e31b44b2"
format_stamp: "Formatted at 2026-06-20 02:28:46 on dist-test-slave-r706"
server_key: "8a6c9012262ffc2998beba0c0741a9ba"
server_key_iv: "ce432737d76a59dbbfd7837181033058"
server_key_version: "encryptionkey@0"
I20260620 02:28:46.384956 4504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:28:46.388803 4518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:46.390391 4504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:46.390568 4504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "4e91dcfc975a4c64a93332b8e31b44b2"
format_stamp: "Formatted at 2026-06-20 02:28:46 on dist-test-slave-r706"
server_key: "8a6c9012262ffc2998beba0c0741a9ba"
server_key_iv: "ce432737d76a59dbbfd7837181033058"
server_key_version: "encryptionkey@0"
I20260620 02:28:46.390765 4504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:46.414364 4504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:46.418856 4504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:46.419114 4504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:46.438937 4504 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45947
I20260620 02:28:46.439054 4569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45947 every 8 connection(s)
I20260620 02:28:46.439990 4504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:46.443905 4570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:46.449596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4504
I20260620 02:28:46.449848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:46.450634 388 external_mini_cluster.cc:1428] Setting key a046ba380c05d603b29490262d6b8390
I20260620 02:28:46.455796 4570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2: Bootstrap starting.
I20260620 02:28:46.459621 4570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:46.460675 4570 log.cc:824] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:46.463924 4570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2: No bootstrap required, opened a new log
Jun 20 02:28:46 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:46.472155 4570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260620 02:28:46.472586 4570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:46.472718 4570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e91dcfc975a4c64a93332b8e31b44b2, State: Initialized, Role: FOLLOWER
I20260620 02:28:46.473408 4570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260620 02:28:46.473672 4570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:46.473821 4570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:46.474022 4570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:46.474349 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:46.452891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.97.62:45947)
0620 02:28:46.453848 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:46.453861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:46.453911 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:28:46.455603 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:46.455626 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:46.456097 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:46.456432 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:46.458396 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:46.459754 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:46.461183 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:46.462005 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:46.462208 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:46.466348 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:46.466356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:46.466371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:46.466411 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:46.469955 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:46.470703 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:46.470710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:46.470717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:46.470906 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:46.471316 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:46.471320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:46.471326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:46.471868 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:46.472171 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:46.472609 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 02:28:46.473122 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":426,"threads_started":1}
I20260620 02:28:46.477087 4570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260620 02:28:46.477643 4570 leader_election.cc:302] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e91dcfc975a4c64a93332b8e31b44b2; no voters:
I20260620 02:28:46.478291 4570 leader_election.cc:288] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:46.478524 4575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:46.479832 4575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e91dcfc975a4c64a93332b8e31b44b2, State: Running, Role: LEADER
I20260620 02:28:46.480525 4570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:46.480525 4575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260620 02:28:46.483578 4577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e91dcfc975a4c64a93332b8e31b44b2. Latest consensus state: current_term: 1 leader_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } } }
I20260620 02:28:46.484077 4577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:46.484874 4576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e91dcfc975a4c64a93332b8e31b44b2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } } }
I20260620 02:28:46.485240 4576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:46.486774 4584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:46.492539 4584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:46.502265 4584 catalog_manager.cc:1347] Generated new cluster ID: 9906afe8e5b14bbeb1a7e53674f7806c
I20260620 02:28:46.502461 4584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:46.524199 4584 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:46.525820 4584 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:46.536759 4584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2: Generated new TSK 0
I20260620 02:28:46.537683 4584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:46.605533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:46.944094 4596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:46.944478 4596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:46.944664 4596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:46.952041 4596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:46.952227 4596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:46.952435 4596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:46.964964 4596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:46.966436 4596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:46.967790 4596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:46.981611 4602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:46.981742 4604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:46.981822 4601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:46.983306 4596 server_base.cc:1034] running on GCE node
I20260620 02:28:46.984052 4596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:46.985136 4596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:46.986320 4596 hybrid_clock.cc:648] HybridClock initialized: now 1781922526986277 us; error 39 us; skew 500 ppm
I20260620 02:28:46.986644 4596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:46 dist-test-slave-r706 krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:46.989878 4596 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:46.991075 4596 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:46.991214 4596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:46.991560 4596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:46.994788 4596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "44a91a2fb3004db58208f7cd3c1c153b"
format_stamp: "Formatted at 2026-06-20 02:28:46 on dist-test-slave-r706"
server_key: "991ce4b28f4a6139cd0c622516df1a9e"
server_key_iv: "410b7c8b3bd13fb522047a97783f8857"
server_key_version: "encryptionkey@0"
I20260620 02:28:46.995661 4596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "44a91a2fb3004db58208f7cd3c1c153b"
format_stamp: "Formatted at 2026-06-20 02:28:46 on dist-test-slave-r706"
server_key: "991ce4b28f4a6139cd0c622516df1a9e"
server_key_iv: "410b7c8b3bd13fb522047a97783f8857"
server_key_version: "encryptionkey@0"
I20260620 02:28:47.000820 4596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:28:47.004575 4610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:47.006088 4596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:47.006292 4596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "44a91a2fb3004db58208f7cd3c1c153b"
format_stamp: "Formatted at 2026-06-20 02:28:46 on dist-test-slave-r706"
server_key: "991ce4b28f4a6139cd0c622516df1a9e"
server_key_iv: "410b7c8b3bd13fb522047a97783f8857"
server_key_version: "encryptionkey@0"
I20260620 02:28:47.006529 4596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:47.032671 4596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:47.037858 4596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:47.038167 4596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:47.039228 4596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:47.040649 4596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:47.040786 4596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:47.040921 4596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:47.041033 4596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:47.075191 4596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33567
I20260620 02:28:47.075284 4722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33567 every 8 connection(s)
I20260620 02:28:47.076076 4596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:47.077632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4596
I20260620 02:28:47.077884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:47.078624 388 external_mini_cluster.cc:1428] Setting key b336ce98a5604b13e726480f3cf530b4
Jun 20 02:28:47 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:28:47.101086 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:47.079081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47281 (local address 127.0.97.62:45947)
0620 02:28:47.079575 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:47.079580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:47.080724 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 02:28:47.082937 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:47.082942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:47.083028 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:47.083183 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:47.084980 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.085760 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.088030 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.088814 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.088967 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:47.092584 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:47.092593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:47.092601 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:47.092649 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:47.096609 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:47.097853 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:47.097862 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:47.097868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:47.098039 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:47.098854 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:47.098861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:47.098866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:47.099255 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:47.099443 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:47.100259 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 02:28:47.100629 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260620 02:28:47.101886 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:47.079650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45947 (local address 127.0.97.1:47281)
0620 02:28:47.080540 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:47.080596 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:47.082623 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:47.083443 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:47.083451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:47.084112 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:47.084808 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:47.084829 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.085932 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.085945 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:47.087866 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:47.087880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.089128 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.089135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:47.089362 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:47.090087 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:47.090110 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:47.092328 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:47.096865 (+ 4537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:47.096878 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:47.096898 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:47.097680 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:47.098208 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:47.098214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:47.098218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:47.098694 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:47.099440 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:47.099448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:47.099829 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 02:28:47.100091 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 02:28:47.100557 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":338,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:28:47.103741 4723 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45947
I20260620 02:28:47.104135 4723 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:47.105077 4723 heartbeater.cc:511] Master 127.0.97.62:45947 requested a full tablet report, sending...
I20260620 02:28:47.107599 4535 ts_manager.cc:194] Registered new tserver with Master: 44a91a2fb3004db58208f7cd3c1c153b (127.0.97.1:33567)
I20260620 02:28:47.109177 4535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:47.132958 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:47.462826 4729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:47.463109 4729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:47.463210 4729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:47.470291 4729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:47.470408 4729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:47.470532 4729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:47.480885 4729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:47.482427 4729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:47.483768 4729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:47.497484 4734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:47.497399 4735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:47.497949 4737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:47.498232 4729 server_base.cc:1034] running on GCE node
I20260620 02:28:47.498907 4729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:47.499934 4729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:47.501148 4729 hybrid_clock.cc:648] HybridClock initialized: now 1781922527501073 us; error 66 us; skew 500 ppm
I20260620 02:28:47.501552 4729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:47 dist-test-slave-r706 krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:47.505048 4729 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:47.506368 4729 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:47.506494 4729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:47.506858 4729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:47.510085 4729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "b66d46c685e54fcab403d49f20310ef5"
format_stamp: "Formatted at 2026-06-20 02:28:47 on dist-test-slave-r706"
server_key: "db70764b2adc8f782a792a12e8e78f1f"
server_key_iv: "6d65e5417a64156497ee703e3b47d06e"
server_key_version: "encryptionkey@0"
I20260620 02:28:47.510996 4729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b66d46c685e54fcab403d49f20310ef5"
format_stamp: "Formatted at 2026-06-20 02:28:47 on dist-test-slave-r706"
server_key: "db70764b2adc8f782a792a12e8e78f1f"
server_key_iv: "6d65e5417a64156497ee703e3b47d06e"
server_key_version: "encryptionkey@0"
I20260620 02:28:47.516314 4729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:28:47.520185 4743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:47.521816 4729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:47.522010 4729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "b66d46c685e54fcab403d49f20310ef5"
format_stamp: "Formatted at 2026-06-20 02:28:47 on dist-test-slave-r706"
server_key: "db70764b2adc8f782a792a12e8e78f1f"
server_key_iv: "6d65e5417a64156497ee703e3b47d06e"
server_key_version: "encryptionkey@0"
I20260620 02:28:47.522254 4729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:47.552397 4729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:47.556591 4729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:47.556912 4729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:47.557961 4729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:47.559309 4729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:47.559442 4729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:47.559587 4729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:47.559679 4729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:47.593796 4729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34047
I20260620 02:28:47.593894 4855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34047 every 8 connection(s)
I20260620 02:28:47.594708 4729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:47.604852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4729
I20260620 02:28:47.605095 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:47.605664 388 external_mini_cluster.cc:1428] Setting key f15a5c6100f6a55200530038c2cda535
Jun 20 02:28:47 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:47.616107 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:47.597119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42263 (local address 127.0.97.62:45947)
0620 02:28:47.597309 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:47.597323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:47.598526 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 02:28:47.600264 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:47.600268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:47.600338 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:47.600439 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:47.602153 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.603039 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.604713 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.605278 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.605388 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:47.609024 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:47.609034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:47.609043 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:47.609091 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:47.612852 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:47.613674 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:47.613682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:47.613689 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:47.613851 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:47.614258 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:47.614266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:47.614272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:47.614606 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:47.614725 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:47.615678 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 02:28:47.615858 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:28:47.617699 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:47.597638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45947 (local address 127.0.97.2:42263)
0620 02:28:47.598350 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:47.598387 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:47.600006 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:47.600738 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:47.600747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:47.601366 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:47.601950 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:47.601968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.603187 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.603191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:47.604560 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:47.604577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:47.605501 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:47.605508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:47.605795 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:47.606551 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:47.606573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:47.608760 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:47.613054 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:47.613062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:47.613076 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:47.613551 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:47.613961 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:47.613964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:47.613967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:47.614134 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:47.614737 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:47.614747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:47.615233 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 02:28:47.615599 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 02:28:47.616183 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":295,"threads_started":1}
I20260620 02:28:47.619271 4856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45947
I20260620 02:28:47.619621 4856 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:47.620321 4856 heartbeater.cc:511] Master 127.0.97.62:45947 requested a full tablet report, sending...
I20260620 02:28:47.622216 4535 ts_manager.cc:194] Registered new tserver with Master: b66d46c685e54fcab403d49f20310ef5 (127.0.97.2:34047)
I20260620 02:28:47.623064 4535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42263
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:47.657846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:47.996629 4861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:47.996932 4861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:47.997120 4861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:48.004848 4861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:48.004987 4861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:48.005199 4861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:48.015686 4861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:48.017146 4861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:48.018517 4861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:48.033622 4867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:48.033706 4869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:48.033648 4866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:48.034436 4861 server_base.cc:1034] running on GCE node
I20260620 02:28:48.035197 4861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:48.036164 4861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:48.037441 4861 hybrid_clock.cc:648] HybridClock initialized: now 1781922528037404 us; error 76 us; skew 500 ppm
I20260620 02:28:48.037812 4861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:48 dist-test-slave-r706 krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:48.041276 4861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:48.042546 4861 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:48.042696 4861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:48.043057 4861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:48.046279 4861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "dac0c19527da441e94e8b1885b27476b"
format_stamp: "Formatted at 2026-06-20 02:28:48 on dist-test-slave-r706"
server_key: "f8770d13d69779478266120821423a97"
server_key_iv: "d1aef6aaf6835bd65cefd7c9fd22b80f"
server_key_version: "encryptionkey@0"
I20260620 02:28:48.047223 4861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dac0c19527da441e94e8b1885b27476b"
format_stamp: "Formatted at 2026-06-20 02:28:48 on dist-test-slave-r706"
server_key: "f8770d13d69779478266120821423a97"
server_key_iv: "d1aef6aaf6835bd65cefd7c9fd22b80f"
server_key_version: "encryptionkey@0"
I20260620 02:28:48.053412 4861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:28:48.057174 4875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:48.058697 4861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:48.058902 4861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "dac0c19527da441e94e8b1885b27476b"
format_stamp: "Formatted at 2026-06-20 02:28:48 on dist-test-slave-r706"
server_key: "f8770d13d69779478266120821423a97"
server_key_iv: "d1aef6aaf6835bd65cefd7c9fd22b80f"
server_key_version: "encryptionkey@0"
I20260620 02:28:48.059144 4861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:48.078733 4861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:48.083292 4861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:48.083592 4861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:48.084599 4861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:48.086068 4861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:48.086196 4861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:48.086333 4861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:48.086426 4861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:48.112435 4723 heartbeater.cc:503] Master 127.0.97.62:45947 was elected leader, sending a full tablet report...
I20260620 02:28:48.128899 4861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36783
I20260620 02:28:48.128957 4987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36783 every 8 connection(s)
I20260620 02:28:48.129868 4861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:48.139775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 4861
I20260620 02:28:48.140008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:48.140550 388 external_mini_cluster.cc:1428] Setting key d25d2739fcbd536da84c38220b6810bd
Jun 20 02:28:48 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:48.152302 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.132398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59817 (local address 127.0.97.62:45947)
0620 02:28:48.132913 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.132920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.134023 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 02:28:48.135894 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.135899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.135970 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.136071 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:48.137729 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.138745 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.140965 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.141672 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.141829 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.146304 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:48.146309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:48.146313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:48.146350 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:48.149167 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.150027 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.150032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.150034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.150167 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.150592 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.150595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.150597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.150906 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:48.151024 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.151821 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.152013 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
I20260620 02:28:48.153424 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.132938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45947 (local address 127.0.97.3:59817)
0620 02:28:48.133836 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:48.133884 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:48.135603 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:48.136273 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:48.136281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:48.136829 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:48.137528 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.137548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.138900 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.138904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.140802 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.140818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.141903 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.141908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.142246 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.143144 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:48.143180 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:48.146074 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:48.149337 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:48.149346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:48.149361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:48.149865 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:48.150274 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:48.150278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:48.150280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:48.150471 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:48.151054 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:48.151062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:48.151429 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 02:28:48.151675 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:28:48.152113 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":359,"threads_started":1}
I20260620 02:28:48.155069 4988 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45947
I20260620 02:28:48.155495 4988 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:48.156368 4988 heartbeater.cc:511] Master 127.0.97.62:45947 requested a full tablet report, sending...
I20260620 02:28:48.157894 4535 ts_manager.cc:194] Registered new tserver with Master: dac0c19527da441e94e8b1885b27476b (127.0.97.3:36783)
I20260620 02:28:48.158712 4535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59817
I20260620 02:28:48.169175 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:28:48.187819 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.174519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.97.62:45947)
0620 02:28:48.174716 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.174721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.175097 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:28:48.175243 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.175246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.175313 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.175409 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:48.177047 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.177790 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.179056 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.179502 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.179612 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.182208 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:48.182214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:48.182217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:48.182266 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:48.185246 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.185950 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.185954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.185956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.186077 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.186468 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.186471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.186473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.186794 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:48.186976 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.187392 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.187568 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:28:48.192569 4535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:48.200580 4535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:48.231782 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.214744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.97.2:34047)
0620 02:28:48.222244 (+ 7500us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.222257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.222295 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:28:48.222430 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.222442 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.222774 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.223822 (+ 1048us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:48.224082 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.226421 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.228951 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.230515 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.230665 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.230889 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.231269 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.231448 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7307,"thread_start_us":7158,"threads_started":1}
I20260620 02:28:48.231886 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.214088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34047 (local address 127.0.0.1:60248)
0620 02:28:48.215832 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:48.215861 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:48.215992 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:48.223071 (+ 7079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:48.223081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:48.223130 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:48.223574 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.223588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.226572 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.226578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.228746 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.228760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.230733 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.230740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.230960 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.231029 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 02:28:48.231181 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 02:28:48.231401 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1586,"mutex_wait_us":101,"thread_start_us":840,"threads_started":1}
I20260620 02:28:48.232316 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.214913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.0.97.3:36783)
0620 02:28:48.223434 (+ 8521us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.223443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.223481 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:28:48.223580 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.223587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.223870 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.224079 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:48.224916 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.226680 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.229368 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.230554 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.230722 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.230941 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.231751 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.231937 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8325,"thread_start_us":8148,"threads_started":1}
I20260620 02:28:48.232664 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.214410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36783 (local address 127.0.0.1:52846)
0620 02:28:48.222621 (+ 8211us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:48.222647 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:48.222788 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:48.224122 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:48.224130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:48.224174 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:48.224675 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.224688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.226841 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.226846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.229168 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.229180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.231083 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.231088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.231270 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.231302 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:28:48.231380 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 02:28:48.231483 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8083,"mutex_wait_us":33,"thread_start_us":1981,"threads_started":1}
I20260620 02:28:48.233165 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.213837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33567 (local address 127.0.0.1:35120)
0620 02:28:48.214994 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:48.215052 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:48.215237 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:48.225514 (+ 10277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:48.225519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:48.225547 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:48.226029 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.226041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.228391 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.228396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.230537 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.230552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.232401 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.232409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.232580 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.232615 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:28:48.232684 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 02:28:48.232815 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":8490,"threads_started":1}
I20260620 02:28:48.233745 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.214249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.97.1:33567)
0620 02:28:48.224795 (+ 10546us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.224803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.224832 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:28:48.224931 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.224937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.225190 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.226231 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:48.226608 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.228229 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.230723 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.232148 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.233075 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.233316 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.233393 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.233511 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10359,"thread_start_us":10219,"threads_started":1}
I20260620 02:28:48.236557 4791 tablet_service.cc:1467] Processing CreateTablet for tablet a1cb04fab29f4c28910855c2468de74e (DEFAULT_TABLE table=test-table [id=d5558bc397274adeaea42a672b87a7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:48.237076 4658 tablet_service.cc:1467] Processing CreateTablet for tablet a1cb04fab29f4c28910855c2468de74e (DEFAULT_TABLE table=test-table [id=d5558bc397274adeaea42a672b87a7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:48.237391 4923 tablet_service.cc:1467] Processing CreateTablet for tablet a1cb04fab29f4c28910855c2468de74e (DEFAULT_TABLE table=test-table [id=d5558bc397274adeaea42a672b87a7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:48.238225 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cb04fab29f4c28910855c2468de74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:48.238420 4658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cb04fab29f4c28910855c2468de74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:48.238767 4923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1cb04fab29f4c28910855c2468de74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:48.254027 5008 tablet_bootstrap.cc:492] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Bootstrap starting.
I20260620 02:28:48.254948 5007 tablet_bootstrap.cc:492] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Bootstrap starting.
I20260620 02:28:48.259101 5008 tablet_bootstrap.cc:654] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:48.259094 5007 tablet_bootstrap.cc:654] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:48.260181 5009 tablet_bootstrap.cc:492] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Bootstrap starting.
I20260620 02:28:48.260663 5007 log.cc:824] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:48.260705 5008 log.cc:824] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:48.264074 5008 tablet_bootstrap.cc:492] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: No bootstrap required, opened a new log
I20260620 02:28:48.264263 5007 tablet_bootstrap.cc:492] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: No bootstrap required, opened a new log
I20260620 02:28:48.264430 5008 ts_tablet_manager.cc:1397] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 02:28:48.264531 5009 tablet_bootstrap.cc:654] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:48.264559 5007 ts_tablet_manager.cc:1397] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 02:28:48.266042 5009 log.cc:824] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:48.270402 5009 tablet_bootstrap.cc:492] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: No bootstrap required, opened a new log
I20260620 02:28:48.270700 5009 ts_tablet_manager.cc:1397] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 02:28:48.273561 5008 raft_consensus.cc:348] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.273964 5008 raft_consensus.cc:374] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:48.274134 5008 raft_consensus.cc:729] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66d46c685e54fcab403d49f20310ef5, State: Initialized, Role: FOLLOWER
I20260620 02:28:48.274782 5008 consensus_queue.cc:260] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.275315 5007 raft_consensus.cc:348] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.275794 5007 raft_consensus.cc:374] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:48.275938 5007 raft_consensus.cc:729] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a91a2fb3004db58208f7cd3c1c153b, State: Initialized, Role: FOLLOWER
I20260620 02:28:48.276096 4856 heartbeater.cc:503] Master 127.0.97.62:45947 was elected leader, sending a full tablet report...
I20260620 02:28:48.276541 5007 consensus_queue.cc:260] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.278434 5007 ts_tablet_manager.cc:1428] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260620 02:28:48.279583 5008 ts_tablet_manager.cc:1428] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260620 02:28:48.281975 5009 raft_consensus.cc:348] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.282444 5009 raft_consensus.cc:374] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:48.282575 5009 raft_consensus.cc:729] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac0c19527da441e94e8b1885b27476b, State: Initialized, Role: FOLLOWER
I20260620 02:28:48.283205 5009 consensus_queue.cc:260] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.284399 4988 heartbeater.cc:503] Master 127.0.97.62:45947 was elected leader, sending a full tablet report...
I20260620 02:28:48.284960 5009 ts_tablet_manager.cc:1428] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
W20260620 02:28:48.328680 4724 tablet.cc:2369] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:48.346742 4857 tablet.cc:2369] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:48.381439 4989 tablet.cc:2369] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:48.491641 5013 raft_consensus.cc:482] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:48.491938 5013 raft_consensus.cc:504] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.493870 5013 leader_election.cc:288] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a91a2fb3004db58208f7cd3c1c153b (127.0.97.1:33567), dac0c19527da441e94e8b1885b27476b (127.0.97.3:36783)
I20260620 02:28:48.501332 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.494363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38703 (local address 127.0.97.1:33567)
0620 02:28:48.494585 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.494592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.495605 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 02:28:48.495866 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.495872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.495951 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.496052 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:48.496773 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.497579 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.499487 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.500275 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.500437 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.500565 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.500883 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.501033 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:28:48.501353 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.494460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57623 (local address 127.0.97.3:36783)
0620 02:28:48.494789 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.494797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.495623 (+ 826us) server_negotiation.cc:407] Connection header received
0620 02:28:48.495868 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.495875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.495939 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.496040 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:28:48.496603 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.497481 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.499420 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.500283 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.500458 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.500582 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.500925 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.501074 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 02:28:48.501483 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.494313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33567 (local address 127.0.97.2:38703)
0620 02:28:48.495478 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:48.495495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:48.495671 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:48.496112 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:48.496117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:48.496144 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:48.496629 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.496637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.497737 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.497743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.499327 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.499338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.500527 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.500533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.500716 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.500748 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:28:48.500856 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:28:48.500986 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":3,"thread_start_us":559,"threads_started":1}
I20260620 02:28:48.501535 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.494368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36783 (local address 127.0.97.2:57623)
0620 02:28:48.495516 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:48.495532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:48.495710 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:48.496054 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:48.496059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:48.496081 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:28:48.496444 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.496453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.497607 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.497615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.499246 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:48.499258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.500591 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.500600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:48.500766 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.500789 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:28:48.500898 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 02:28:48.501041 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":40,"thread_start_us":610,"threads_started":1}
I20260620 02:28:48.502521 4943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cb04fab29f4c28910855c2468de74e" candidate_uuid: "b66d46c685e54fcab403d49f20310ef5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac0c19527da441e94e8b1885b27476b" is_pre_election: true
I20260620 02:28:48.502521 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cb04fab29f4c28910855c2468de74e" candidate_uuid: "b66d46c685e54fcab403d49f20310ef5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a91a2fb3004db58208f7cd3c1c153b" is_pre_election: true
I20260620 02:28:48.503031 4943 raft_consensus.cc:2454] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66d46c685e54fcab403d49f20310ef5 in term 0.
I20260620 02:28:48.503039 4678 raft_consensus.cc:2454] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66d46c685e54fcab403d49f20310ef5 in term 0.
I20260620 02:28:48.503679 4747 leader_election.cc:302] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66d46c685e54fcab403d49f20310ef5, dac0c19527da441e94e8b1885b27476b; no voters:
I20260620 02:28:48.504086 5013 raft_consensus.cc:2781] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:48.504245 5013 raft_consensus.cc:482] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:48.504365 5013 raft_consensus.cc:3037] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:48.506577 5013 raft_consensus.cc:504] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.507169 5013 leader_election.cc:288] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [CANDIDATE]: Term 1 election: Requested vote from peers 44a91a2fb3004db58208f7cd3c1c153b (127.0.97.1:33567), dac0c19527da441e94e8b1885b27476b (127.0.97.3:36783)
I20260620 02:28:48.507627 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cb04fab29f4c28910855c2468de74e" candidate_uuid: "b66d46c685e54fcab403d49f20310ef5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a91a2fb3004db58208f7cd3c1c153b"
I20260620 02:28:48.507627 4943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cb04fab29f4c28910855c2468de74e" candidate_uuid: "b66d46c685e54fcab403d49f20310ef5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac0c19527da441e94e8b1885b27476b"
I20260620 02:28:48.507839 4943 raft_consensus.cc:3037] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:48.507839 4678 raft_consensus.cc:3037] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:48.511089 4678 raft_consensus.cc:2454] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66d46c685e54fcab403d49f20310ef5 in term 1.
I20260620 02:28:48.511089 4943 raft_consensus.cc:2454] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66d46c685e54fcab403d49f20310ef5 in term 1.
I20260620 02:28:48.511533 4747 leader_election.cc:302] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66d46c685e54fcab403d49f20310ef5, dac0c19527da441e94e8b1885b27476b; no voters:
I20260620 02:28:48.511881 5013 raft_consensus.cc:2781] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:48.512390 5013 raft_consensus.cc:686] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 1 LEADER]: Becoming Leader. State: Replica: b66d46c685e54fcab403d49f20310ef5, State: Running, Role: LEADER
I20260620 02:28:48.512990 5013 consensus_queue.cc:237] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } }
I20260620 02:28:48.517822 4534 catalog_manager.cc:5495] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b66d46c685e54fcab403d49f20310ef5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b66d46c685e54fcab403d49f20310ef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66d46c685e54fcab403d49f20310ef5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 } health_report { overall_health: UNKNOWN } } }
I20260620 02:28:48.600087 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.593495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.0.97.2:34047)
0620 02:28:48.593755 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.593759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.593830 (+ 71us) server_negotiation.cc:407] Connection header received
0620 02:28:48.594037 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.594041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.594123 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.594257 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:28:48.594665 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.595552 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.597578 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.598154 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.598262 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.598825 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:28:48.599325 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:28:48.599432 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.599641 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.599720 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:28:48.609488 4678 raft_consensus.cc:1261] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 1 FOLLOWER]: Refusing update from remote peer b66d46c685e54fcab403d49f20310ef5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:48.609485 4943 raft_consensus.cc:1261] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 1 FOLLOWER]: Refusing update from remote peer b66d46c685e54fcab403d49f20310ef5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:48.610558 5018 consensus_queue.cc:1035] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a91a2fb3004db58208f7cd3c1c153b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:48.610977 5013 consensus_queue.cc:1035] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac0c19527da441e94e8b1885b27476b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:48.624897 5021 mvcc.cc:205] Tried to move back new op lower bound from 7298754677173051392 to 7298754676793651200. Current Snapshot: MvccSnapshot[applied={T|T < 7298754677173051392}]
I20260620 02:28:48.642967 4535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:48.643301 4535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:48.647202 4535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e91dcfc975a4c64a93332b8e31b44b2: Sending DeleteTablet for 3 replicas of tablet a1cb04fab29f4c28910855c2468de74e
I20260620 02:28:48.648538 4658 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cb04fab29f4c28910855c2468de74e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35120
I20260620 02:28:48.648582 4923 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cb04fab29f4c28910855c2468de74e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52846
I20260620 02:28:48.649637 5030 tablet_replica.cc:330] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: stopping tablet replica
I20260620 02:28:48.649699 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cb04fab29f4c28910855c2468de74e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60248
I20260620 02:28:48.649899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4596
I20260620 02:28:48.650127 5030 raft_consensus.cc:2229] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:48.650722 5030 raft_consensus.cc:2258] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:48.651513 5031 tablet_replica.cc:330] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: stopping tablet replica
I20260620 02:28:48.651948 5031 raft_consensus.cc:2229] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:48.652429 5031 raft_consensus.cc:2258] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:48.655267 5031 ts_tablet_manager.cc:1905] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:48.664335 5031 ts_tablet_manager.cc:1918] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:48.664557 5031 log.cc:1196] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/a1cb04fab29f4c28910855c2468de74e
I20260620 02:28:48.665222 5031 ts_tablet_manager.cc:1939] T a1cb04fab29f4c28910855c2468de74e P 44a91a2fb3004db58208f7cd3c1c153b: Deleting consensus metadata
I20260620 02:28:48.665257 5030 ts_tablet_manager.cc:1905] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:48.667266 4521 catalog_manager.cc:4841] TS 44a91a2fb3004db58208f7cd3c1c153b (127.0.97.1:33567): tablet a1cb04fab29f4c28910855c2468de74e (table test-table [id=d5558bc397274adeaea42a672b87a7bb]) successfully deleted
I20260620 02:28:48.670791 5030 ts_tablet_manager.cc:1918] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:48.671003 5030 log.cc:1196] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/a1cb04fab29f4c28910855c2468de74e
I20260620 02:28:48.671622 5030 ts_tablet_manager.cc:1939] T a1cb04fab29f4c28910855c2468de74e P dac0c19527da441e94e8b1885b27476b: Deleting consensus metadata
I20260620 02:28:48.672415 5032 tablet_replica.cc:330] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: stopping tablet replica
I20260620 02:28:48.672950 5032 raft_consensus.cc:2229] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:48.673714 4522 catalog_manager.cc:4841] TS dac0c19527da441e94e8b1885b27476b (127.0.97.3:36783): tablet a1cb04fab29f4c28910855c2468de74e (table test-table [id=d5558bc397274adeaea42a672b87a7bb]) successfully deleted
I20260620 02:28:48.673900 5032 raft_consensus.cc:2258] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:48.675918 5032 ts_tablet_manager.cc:1905] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:28:48 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:48.682746 5032 ts_tablet_manager.cc:1918] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:48.682967 5032 log.cc:1196] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/a1cb04fab29f4c28910855c2468de74e
I20260620 02:28:48.683580 5032 ts_tablet_manager.cc:1939] T a1cb04fab29f4c28910855c2468de74e P b66d46c685e54fcab403d49f20310ef5: Deleting consensus metadata
I20260620 02:28:48.685460 4519 catalog_manager.cc:4841] TS b66d46c685e54fcab403d49f20310ef5 (127.0.97.2:34047): tablet a1cb04fab29f4c28910855c2468de74e (table test-table [id=d5558bc397274adeaea42a672b87a7bb]) successfully deleted
I20260620 02:28:48.690438 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.651309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.0.97.1:33567)
0620 02:28:48.651500 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.651507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.661522 (+ 10015us) server_negotiation.cc:407] Connection header received
0620 02:28:48.661814 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.661820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.661900 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.661983 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:48.673121 (+ 11138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.674361 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.676430 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.677103 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.677257 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.682879 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:48.682900 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:48.682920 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:48.682979 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:48.687221 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.688048 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.688058 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.688062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.688252 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.688711 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.688716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.688720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.689223 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:48.689555 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.689805 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.690048 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:28:48.691170 4718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:48.782411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4596
I20260620 02:28:48.800658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4729
Jun 20 02:28:48 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:28:48.819296 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.801497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.97.2:34047)
0620 02:28:48.801666 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.801670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.801714 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:28:48.801916 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.801920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.801994 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.802081 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:48.803737 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.804502 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.806487 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.806947 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.807053 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.812122 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:48.812144 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:48.812159 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:48.812207 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:48.816166 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.816910 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.816919 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.816923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.817136 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.817583 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.817588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.817592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.818058 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:48.818405 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.818635 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.818877 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:28:48.820063 4851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:48.935897 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4729
I20260620 02:28:48.955300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4861
Jun 20 02:28:48 dist-test-slave-r706 krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:28:48.973296 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:48.956349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.0.97.3:36783)
0620 02:28:48.956627 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:48.956634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:48.956655 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:28:48.956742 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:48.956748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:48.956836 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:48.956965 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:48.958709 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.959741 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.961705 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:48.962180 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:48.962333 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:48.967391 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:48.967408 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:48.967425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:48.967466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:48.970309 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.971018 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.971026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.971028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.971158 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:48.971537 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:48.971541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:48.971543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:48.972021 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:48.972346 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:48.972663 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:28:48.972885 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:28:48.974129 4983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:49.063548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4861
I20260620 02:28:49.080593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 4504
I20260620 02:28:49.081854 4565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:49.194340 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 4504
2026-06-20T02:28:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4536 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T02:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:49Z Disabled control of system clock
I20260620 02:28:49.267216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35673
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:28:49.594048 5041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:49.594368 5041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:49.594475 5041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:49.601863 5041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:49.601986 5041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:49.602026 5041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:49.602075 5041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:49.602121 5041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:49.611972 5041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:49.613315 5041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:49.615207 5041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:49.628008 5047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:49.628185 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:49.628516 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:49.629201 5041 server_base.cc:1034] running on GCE node
I20260620 02:28:49.630074 5041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:49.631559 5041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:49.632812 5041 hybrid_clock.cc:648] HybridClock initialized: now 1781922529632774 us; error 53 us; skew 500 ppm
I20260620 02:28:49.633258 5041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:49.640201 5041 webserver.cc:460] Webserver started at http://127.0.97.62:39799/ using document root <none> and password file <none>
I20260620 02:28:49.641103 5041 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:49.641252 5041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:49.641630 5041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:49.644742 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "161b4ac0a1824e51a262dd68f30245a2"
format_stamp: "Formatted at 2026-06-20 02:28:49 on dist-test-slave-r706"
server_key: "f21ee558cabcf9db27099f0c7d8f80ee"
server_key_iv: "1c6914bd9533ff34a49d34dedc3fa42d"
server_key_version: "encryptionkey@0"
I20260620 02:28:49.645642 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "161b4ac0a1824e51a262dd68f30245a2"
format_stamp: "Formatted at 2026-06-20 02:28:49 on dist-test-slave-r706"
server_key: "f21ee558cabcf9db27099f0c7d8f80ee"
server_key_iv: "1c6914bd9533ff34a49d34dedc3fa42d"
server_key_version: "encryptionkey@0"
I20260620 02:28:49.651012 5041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:28:49.654518 5055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:49.655827 5041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:28:49.656015 5041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "161b4ac0a1824e51a262dd68f30245a2"
format_stamp: "Formatted at 2026-06-20 02:28:49 on dist-test-slave-r706"
server_key: "f21ee558cabcf9db27099f0c7d8f80ee"
server_key_iv: "1c6914bd9533ff34a49d34dedc3fa42d"
server_key_version: "encryptionkey@0"
I20260620 02:28:49.656252 5041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:49.662097 5041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:49.662925 5041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:49.663240 5041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:49.682891 5041 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35673
I20260620 02:28:49.682932 5106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35673 every 8 connection(s)
I20260620 02:28:49.684386 5041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:49.687857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5041
I20260620 02:28:49.688069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:49.688597 5107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:49.688853 388 external_mini_cluster.cc:1428] Setting key d834cf72e096d3f10d23b52657a5aac4
I20260620 02:28:49.698334 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:49.691186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.0.97.62:35673)
0620 02:28:49.692358 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:49.692373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:49.692424 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:28:49.694174 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:49.694205 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:49.694768 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:49.695242 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:49.696293 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:49.696306 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:49.696324 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:49.696345 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:49.696415 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:49.696446 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:49.696669 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:49.696817 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 02:28:49.697147 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":392,"threads_started":1}
I20260620 02:28:49.703179 5107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2: Bootstrap starting.
I20260620 02:28:49.706267 5107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:49.707377 5107 log.cc:824] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:49.710194 5107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2: No bootstrap required, opened a new log
I20260620 02:28:49.717368 5107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161b4ac0a1824e51a262dd68f30245a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260620 02:28:49.717777 5107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:49.717922 5107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161b4ac0a1824e51a262dd68f30245a2, State: Initialized, Role: FOLLOWER
I20260620 02:28:49.718648 5107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161b4ac0a1824e51a262dd68f30245a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260620 02:28:49.718868 5107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:49.718968 5107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:49.719094 5107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:49.721289 5107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161b4ac0a1824e51a262dd68f30245a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260620 02:28:49.721740 5107 leader_election.cc:302] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161b4ac0a1824e51a262dd68f30245a2; no voters:
I20260620 02:28:49.722275 5107 leader_election.cc:288] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:49.722445 5112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:49.723150 5112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [term 1 LEADER]: Becoming Leader. State: Replica: 161b4ac0a1824e51a262dd68f30245a2, State: Running, Role: LEADER
I20260620 02:28:49.723668 5112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161b4ac0a1824e51a262dd68f30245a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260620 02:28:49.724403 5107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:49.726652 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161b4ac0a1824e51a262dd68f30245a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161b4ac0a1824e51a262dd68f30245a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } } }
I20260620 02:28:49.727025 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:49.728133 5118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:49.728231 5114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161b4ac0a1824e51a262dd68f30245a2. Latest consensus state: current_term: 1 leader_uuid: "161b4ac0a1824e51a262dd68f30245a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161b4ac0a1824e51a262dd68f30245a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } } }
I20260620 02:28:49.730151 5114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:49.734901 5118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:49.746529 5118 catalog_manager.cc:1347] Generated new cluster ID: 6376ca3111f34e53951220819180a119
I20260620 02:28:49.746655 5118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:49.781311 5118 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:49.781836 5118 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:49.791476 5118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2: Generated new TSK 0
I20260620 02:28:49.792593 5118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:28:49.801774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--builtin_ntp_servers=127.0.97.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:28:50.149745 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:50.150135 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:50.150314 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:50.158335 5131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:50.158746 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:50.158926 5131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:50.168766 5131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:50.170156 5131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:50.171967 5131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:50.185433 5136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:50.185467 5137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:50.185535 5139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:50.186728 5131 server_base.cc:1034] running on GCE node
I20260620 02:28:50.187734 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:50.188704 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:50.189913 5131 hybrid_clock.cc:648] HybridClock initialized: now 1781922530189827 us; error 76 us; skew 500 ppm
I20260620 02:28:50.190258 5131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:50.192610 5131 webserver.cc:460] Webserver started at http://127.0.97.1:41729/ using document root <none> and password file <none>
I20260620 02:28:50.193419 5131 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:50.193511 5131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:50.193888 5131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:50.196800 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "b9ab54de55a14b6191ba59cffb758ad8"
format_stamp: "Formatted at 2026-06-20 02:28:50 on dist-test-slave-r706"
server_key: "68190eda0e898dd95c47d3711c14400f"
server_key_iv: "069671d809f9dfdb5044977038f18115"
server_key_version: "encryptionkey@0"
I20260620 02:28:50.197788 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b9ab54de55a14b6191ba59cffb758ad8"
format_stamp: "Formatted at 2026-06-20 02:28:50 on dist-test-slave-r706"
server_key: "68190eda0e898dd95c47d3711c14400f"
server_key_iv: "069671d809f9dfdb5044977038f18115"
server_key_version: "encryptionkey@0"
I20260620 02:28:50.203184 5131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:50.206779 5145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:50.208354 5131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:28:50.208565 5131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "b9ab54de55a14b6191ba59cffb758ad8"
format_stamp: "Formatted at 2026-06-20 02:28:50 on dist-test-slave-r706"
server_key: "68190eda0e898dd95c47d3711c14400f"
server_key_iv: "069671d809f9dfdb5044977038f18115"
server_key_version: "encryptionkey@0"
I20260620 02:28:50.208810 5131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:50.215118 5131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:50.215993 5131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:50.216282 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:50.217329 5131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:50.218801 5131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:50.218919 5131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:50.219058 5131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:50.219163 5131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:50.255162 5131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39145
I20260620 02:28:50.255281 5257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39145 every 8 connection(s)
I20260620 02:28:50.256536 5131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:50.264178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5131
I20260620 02:28:50.264402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:50.265357 388 external_mini_cluster.cc:1428] Setting key 423324f024a3a7f3766df95b363e6a25
I20260620 02:28:50.265760 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:50.259218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45029 (local address 127.0.97.62:35673)
0620 02:28:50.259714 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:50.259718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:50.261047 (+ 1329us) server_negotiation.cc:407] Connection header received
0620 02:28:50.263140 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:50.263147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:50.263249 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:50.263377 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:50.264916 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:50.264921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:50.264924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:50.264940 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:50.264986 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:50.265052 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:50.265178 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:50.265403 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 02:28:50.265484 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260620 02:28:50.266866 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:50.259857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35673 (local address 127.0.97.1:45029)
0620 02:28:50.260861 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:50.260902 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:50.262614 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:50.263647 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:50.263656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:50.263754 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:50.264543 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:50.264548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:50.264603 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:50.264728 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:50.265169 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:50.265181 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:50.265192 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:28:50.265495 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 02:28:50.265770 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":323,"threads_started":1}
I20260620 02:28:50.268620 5258 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35673
I20260620 02:28:50.268673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--builtin_ntp_servers=127.0.97.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:28:50.268980 5258 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:50.269651 5258 heartbeater.cc:511] Master 127.0.97.62:35673 requested a full tablet report, sending...
I20260620 02:28:50.271927 5072 ts_manager.cc:194] Registered new tserver with Master: b9ab54de55a14b6191ba59cffb758ad8 (127.0.97.1:39145)
W20260620 02:28:50.607758 5262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:50.608052 5262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:50.608213 5262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:50.615698 5262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:50.615808 5262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:50.615988 5262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:50.625346 5262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:50.626753 5262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:50.628582 5262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:50.642171 5270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:50.642318 5267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:50.642293 5268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:50.643170 5262 server_base.cc:1034] running on GCE node
I20260620 02:28:50.643819 5262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:50.644843 5262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:50.646085 5262 hybrid_clock.cc:648] HybridClock initialized: now 1781922530646002 us; error 79 us; skew 500 ppm
I20260620 02:28:50.646430 5262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:50.648839 5262 webserver.cc:460] Webserver started at http://127.0.97.2:42707/ using document root <none> and password file <none>
I20260620 02:28:50.649632 5262 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:50.649780 5262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:50.650128 5262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:50.653172 5262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "acdfa11624824eaaae1d393d87177825"
format_stamp: "Formatted at 2026-06-20 02:28:50 on dist-test-slave-r706"
server_key: "2e39d030d6f97e0faf90b242b4adafe2"
server_key_iv: "066119f4675d95e8d4526b42883e9e30"
server_key_version: "encryptionkey@0"
I20260620 02:28:50.654019 5262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "acdfa11624824eaaae1d393d87177825"
format_stamp: "Formatted at 2026-06-20 02:28:50 on dist-test-slave-r706"
server_key: "2e39d030d6f97e0faf90b242b4adafe2"
server_key_iv: "066119f4675d95e8d4526b42883e9e30"
server_key_version: "encryptionkey@0"
I20260620 02:28:50.659551 5262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:50.663478 5276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:50.664943 5262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:50.665179 5262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "acdfa11624824eaaae1d393d87177825"
format_stamp: "Formatted at 2026-06-20 02:28:50 on dist-test-slave-r706"
server_key: "2e39d030d6f97e0faf90b242b4adafe2"
server_key_iv: "066119f4675d95e8d4526b42883e9e30"
server_key_version: "encryptionkey@0"
I20260620 02:28:50.665416 5262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:50.671242 5262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:50.672013 5262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:50.672259 5262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:50.673214 5262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:50.674572 5262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:50.674657 5262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:50.674741 5262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:50.674795 5262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:50.710084 5262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41465
I20260620 02:28:50.710144 5388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41465 every 8 connection(s)
I20260620 02:28:50.711505 5262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:50.719732 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:50.714328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52849 (local address 127.0.97.62:35673)
0620 02:28:50.714552 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:50.714559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:50.715957 (+ 1398us) server_negotiation.cc:407] Connection header received
0620 02:28:50.717708 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:50.717712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:50.717771 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:50.717862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:50.719126 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:50.719131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:50.719134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:50.719153 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:50.719184 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:50.719196 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:50.719290 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:50.719452 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 02:28:50.719504 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:28:50.720672 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:50.714875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35673 (local address 127.0.97.2:52849)
0620 02:28:50.715791 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:50.715827 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:50.717448 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:50.718080 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:50.718088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:50.718179 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:50.718826 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:50.718828 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:50.718864 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:50.718949 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:50.719277 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:50.719284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:50.719289 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:28:50.719497 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 02:28:50.719694 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":288,"threads_started":1}
I20260620 02:28:50.721099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5262
I20260620 02:28:50.721307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:50.721809 388 external_mini_cluster.cc:1428] Setting key 0413fa1afcd3542585ba98689e8785c8
I20260620 02:28:50.722275 5389 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35673
I20260620 02:28:50.722611 5389 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:50.723253 5389 heartbeater.cc:511] Master 127.0.97.62:35673 requested a full tablet report, sending...
I20260620 02:28:50.724887 5072 ts_manager.cc:194] Registered new tserver with Master: acdfa11624824eaaae1d393d87177825 (127.0.97.2:41465)
I20260620 02:28:50.725059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--builtin_ntp_servers=127.0.97.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:28:51.064860 5392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:51.065284 5392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:51.065476 5392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:51.073333 5392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:51.073470 5392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:51.073684 5392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:51.083460 5392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:51.084869 5392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:51.086864 5392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:51.101583 5398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:51.101739 5397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:51.101745 5400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:51.102892 5392 server_base.cc:1034] running on GCE node
I20260620 02:28:51.103662 5392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:51.104833 5392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:51.106112 5392 hybrid_clock.cc:648] HybridClock initialized: now 1781922531106057 us; error 52 us; skew 500 ppm
I20260620 02:28:51.106534 5392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:51.110419 5392 webserver.cc:460] Webserver started at http://127.0.97.3:33363/ using document root <none> and password file <none>
I20260620 02:28:51.114086 5392 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:51.114264 5392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:51.114683 5392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:51.123329 5392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "3991538cb6494302971d1b2153606935"
format_stamp: "Formatted at 2026-06-20 02:28:51 on dist-test-slave-r706"
server_key: "e91d4626ced31f6b4b836c0c0c074f41"
server_key_iv: "bf1ae733b46b3489d80036b0dfbd4227"
server_key_version: "encryptionkey@0"
I20260620 02:28:51.125241 5392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3991538cb6494302971d1b2153606935"
format_stamp: "Formatted at 2026-06-20 02:28:51 on dist-test-slave-r706"
server_key: "e91d4626ced31f6b4b836c0c0c074f41"
server_key_iv: "bf1ae733b46b3489d80036b0dfbd4227"
server_key_version: "encryptionkey@0"
I20260620 02:28:51.146960 5392 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.010s sys 0.008s
I20260620 02:28:51.153505 5406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:51.154981 5392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:28:51.155203 5392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "3991538cb6494302971d1b2153606935"
format_stamp: "Formatted at 2026-06-20 02:28:51 on dist-test-slave-r706"
server_key: "e91d4626ced31f6b4b836c0c0c074f41"
server_key_iv: "bf1ae733b46b3489d80036b0dfbd4227"
server_key_version: "encryptionkey@0"
I20260620 02:28:51.155462 5392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:51.161507 5392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:51.162357 5392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:51.162662 5392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:51.163985 5392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:51.166152 5392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:51.166280 5392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:51.166409 5392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:51.166489 5392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:51.210072 5392 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39651
I20260620 02:28:51.210170 5518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39651 every 8 connection(s)
I20260620 02:28:51.212051 5392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:51.218283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5392
I20260620 02:28:51.218479 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:51.219055 388 external_mini_cluster.cc:1428] Setting key c3376c0ce4f9354161a94626262d656b
I20260620 02:28:51.223423 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.216620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57603 (local address 127.0.97.62:35673)
0620 02:28:51.216835 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.216845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.217393 (+ 548us) server_negotiation.cc:407] Connection header received
0620 02:28:51.220594 (+ 3201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.220600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.220678 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.220772 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.222619 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.222625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.222629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.222651 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.222693 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.222717 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.222835 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.223052 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.223129 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:28:51.224431 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.215958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35673 (local address 127.0.97.3:57603)
0620 02:28:51.217208 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:51.217271 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:51.219988 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:51.221145 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:51.221158 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:51.221311 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:51.222166 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:51.222170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:51.222231 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:51.222375 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:51.222809 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:51.222820 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:51.222829 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:28:51.223150 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 02:28:51.223421 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":331,"threads_started":1}
I20260620 02:28:51.225982 5519 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35673
I20260620 02:28:51.226372 5519 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:51.227238 5519 heartbeater.cc:511] Master 127.0.97.62:35673 requested a full tablet report, sending...
I20260620 02:28:51.228928 5072 ts_manager.cc:194] Registered new tserver with Master: 3991538cb6494302971d1b2153606935 (127.0.97.3:39651)
I20260620 02:28:51.235064 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:28:51.242467 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.239668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.0.97.62:35673)
0620 02:28:51.239828 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.239832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.240468 (+ 636us) server_negotiation.cc:407] Connection header received
0620 02:28:51.240685 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.240689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.240745 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.240857 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.241789 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.241794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.241799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.241817 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.241858 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.241870 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.241997 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.242134 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.242197 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:28:51.246246 5072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:28:51.253152 5072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:28:51.275159 5258 heartbeater.cc:503] Master 127.0.97.62:35673 was elected leader, sending a full tablet report...
I20260620 02:28:51.297587 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.266129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39651 (local address 127.0.0.1:43258)
0620 02:28:51.291741 (+ 25612us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:51.291807 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:51.292054 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:51.295121 (+ 3067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:51.295129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:51.295224 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:51.296093 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:51.296099 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:51.296141 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:51.296259 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:51.296805 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:51.296820 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:51.296831 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:28:51.297055 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 02:28:51.297155 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25220,"thread_start_us":27426,"threads_started":1}
I20260620 02:28:51.298048 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.266640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.0.97.3:39651)
0620 02:28:51.292851 (+ 26211us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.292863 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.292910 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:28:51.293025 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.293060 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.293406 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.293609 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.296504 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.296518 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.296537 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.296571 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.296648 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.296675 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.296888 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.297402 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.297509 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25980,"thread_start_us":25774,"threads_started":1}
I20260620 02:28:51.298467 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.266722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39145 (local address 127.0.0.1:48990)
0620 02:28:51.294740 (+ 28018us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:51.294756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:51.294858 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:51.295400 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:51.295405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:51.295429 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:51.296224 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:51.296227 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:51.296244 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:51.296289 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:51.297775 (+ 1486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:51.297779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:51.297784 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:28:51.297932 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 02:28:51.297991 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27897,"thread_start_us":28950,"threads_started":1}
I20260620 02:28:51.299374 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.267479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.0.97.1:39145)
0620 02:28:51.290626 (+ 23147us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.290641 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.294968 (+ 4327us) server_negotiation.cc:407] Connection header received
0620 02:28:51.295055 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.295060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.295215 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.295400 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.297408 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.297415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.297427 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.297444 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.297511 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.297532 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.297711 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.298761 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.298861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22839,"thread_start_us":23180,"threads_started":1}
I20260620 02:28:51.300591 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.266234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41465 (local address 127.0.0.1:42564)
0620 02:28:51.293279 (+ 27045us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:51.293305 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:51.293447 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:51.297906 (+ 4459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:51.297911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:51.297935 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:51.298742 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:51.298745 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:51.298762 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:51.298806 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:51.299977 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:51.299981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:51.299986 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:28:51.300117 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:28:51.300171 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26838,"thread_start_us":28269,"threads_started":1}
I20260620 02:28:51.301517 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.266666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.0.97.2:41465)
0620 02:28:51.297375 (+ 30709us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.297383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.297413 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:28:51.297486 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.297492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.297697 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.298477 (+ 780us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.299723 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.299732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.299748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.299775 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.299830 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.299849 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.300984 (+ 1135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.301088 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.301193 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30547,"thread_start_us":30414,"threads_started":1}
I20260620 02:28:51.302517 5454 tablet_service.cc:1467] Processing CreateTablet for tablet febbed8b24ae420fb879799ba8bdc5c1 (DEFAULT_TABLE table=test-table [id=d54ccdfa90304f1eba119423eedd7983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:51.303377 5193 tablet_service.cc:1467] Processing CreateTablet for tablet febbed8b24ae420fb879799ba8bdc5c1 (DEFAULT_TABLE table=test-table [id=d54ccdfa90304f1eba119423eedd7983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:51.304190 5454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet febbed8b24ae420fb879799ba8bdc5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:51.304740 5193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet febbed8b24ae420fb879799ba8bdc5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:51.305104 5324 tablet_service.cc:1467] Processing CreateTablet for tablet febbed8b24ae420fb879799ba8bdc5c1 (DEFAULT_TABLE table=test-table [id=d54ccdfa90304f1eba119423eedd7983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:28:51.306440 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet febbed8b24ae420fb879799ba8bdc5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:51.316608 5537 tablet_bootstrap.cc:492] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Bootstrap starting.
I20260620 02:28:51.320624 5537 tablet_bootstrap.cc:654] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:51.321542 5538 tablet_bootstrap.cc:492] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Bootstrap starting.
I20260620 02:28:51.322055 5537 log.cc:824] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:51.323400 5539 tablet_bootstrap.cc:492] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Bootstrap starting.
I20260620 02:28:51.325688 5538 tablet_bootstrap.cc:654] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:51.327001 5538 log.cc:824] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:51.327206 5539 tablet_bootstrap.cc:654] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:51.327342 5537 tablet_bootstrap.cc:492] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: No bootstrap required, opened a new log
I20260620 02:28:51.327668 5537 ts_tablet_manager.cc:1397] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 02:28:51.328477 5539 log.cc:824] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:51.330242 5538 tablet_bootstrap.cc:492] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: No bootstrap required, opened a new log
I20260620 02:28:51.330507 5538 ts_tablet_manager.cc:1397] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:28:51.331028 5539 tablet_bootstrap.cc:492] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: No bootstrap required, opened a new log
I20260620 02:28:51.331269 5539 ts_tablet_manager.cc:1397] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 02:28:51.338409 5537 raft_consensus.cc:348] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.338912 5537 raft_consensus.cc:374] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:51.339061 5537 raft_consensus.cc:729] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ab54de55a14b6191ba59cffb758ad8, State: Initialized, Role: FOLLOWER
I20260620 02:28:51.339069 5539 raft_consensus.cc:348] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.339597 5539 raft_consensus.cc:374] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:51.339708 5539 raft_consensus.cc:729] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acdfa11624824eaaae1d393d87177825, State: Initialized, Role: FOLLOWER
I20260620 02:28:51.339756 5537 consensus_queue.cc:260] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.340281 5539 consensus_queue.cc:260] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.341063 5538 raft_consensus.cc:348] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.341519 5538 raft_consensus.cc:374] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:51.341691 5538 raft_consensus.cc:729] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3991538cb6494302971d1b2153606935, State: Initialized, Role: FOLLOWER
I20260620 02:28:51.341922 5537 ts_tablet_manager.cc:1428] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 02:28:51.342481 5538 consensus_queue.cc:260] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.342983 5389 heartbeater.cc:503] Master 127.0.97.62:35673 was elected leader, sending a full tablet report...
I20260620 02:28:51.343595 5539 ts_tablet_manager.cc:1428] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260620 02:28:51.344432 5519 heartbeater.cc:503] Master 127.0.97.62:35673 was elected leader, sending a full tablet report...
I20260620 02:28:51.345368 5538 ts_tablet_manager.cc:1428] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 02:28:51.427924 5544 raft_consensus.cc:482] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:28:51.428244 5544 raft_consensus.cc:504] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.430047 5544 leader_election.cc:288] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3991538cb6494302971d1b2153606935 (127.0.97.3:39651), b9ab54de55a14b6191ba59cffb758ad8 (127.0.97.1:39145)
I20260620 02:28:51.433768 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.430435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46117 (local address 127.0.97.3:39651)
0620 02:28:51.430681 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.430690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.431499 (+ 809us) server_negotiation.cc:407] Connection header received
0620 02:28:51.431632 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.431640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.431710 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.431812 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.433029 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.433035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.433040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.433064 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.433109 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.433127 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.433249 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.433392 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.433463 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:28:51.433866 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.430478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52823 (local address 127.0.97.1:39145)
0620 02:28:51.430727 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.430736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.431688 (+ 952us) server_negotiation.cc:407] Connection header received
0620 02:28:51.431846 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.431851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.431909 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.432041 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.433231 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.433237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.433241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.433263 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.433307 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.433329 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.433457 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.433558 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.433624 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 02:28:51.433974 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.430220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39651 (local address 127.0.97.2:46117)
0620 02:28:51.431369 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:51.431389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:51.431507 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:51.431854 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:51.431859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:51.431882 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:51.432747 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:51.432753 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:51.432774 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:51.432829 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:51.433235 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:51.433240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:51.433246 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:28:51.433422 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 02:28:51.433483 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":733,"threads_started":1}
I20260620 02:28:51.434163 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.430365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39145 (local address 127.0.97.2:52823)
0620 02:28:51.431570 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:51.431588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:51.431699 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:51.432118 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:51.432121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:51.432138 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:51.432937 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:28:51.432940 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:51.432958 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:28:51.432998 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:51.433448 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:51.433452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:51.433457 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:28:51.433601 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 02:28:51.433652 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"thread_start_us":654,"threads_started":1}
I20260620 02:28:51.435039 5474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febbed8b24ae420fb879799ba8bdc5c1" candidate_uuid: "acdfa11624824eaaae1d393d87177825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3991538cb6494302971d1b2153606935" is_pre_election: true
I20260620 02:28:51.435279 5213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febbed8b24ae420fb879799ba8bdc5c1" candidate_uuid: "acdfa11624824eaaae1d393d87177825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9ab54de55a14b6191ba59cffb758ad8" is_pre_election: true
I20260620 02:28:51.435698 5474 raft_consensus.cc:2454] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acdfa11624824eaaae1d393d87177825 in term 0.
I20260620 02:28:51.435809 5213 raft_consensus.cc:2454] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acdfa11624824eaaae1d393d87177825 in term 0.
I20260620 02:28:51.436497 5278 leader_election.cc:302] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acdfa11624824eaaae1d393d87177825, b9ab54de55a14b6191ba59cffb758ad8; no voters:
I20260620 02:28:51.437036 5544 raft_consensus.cc:2781] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:28:51.437227 5544 raft_consensus.cc:482] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:28:51.437343 5544 raft_consensus.cc:3037] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:51.439680 5544 raft_consensus.cc:504] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.440339 5544 leader_election.cc:288] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [CANDIDATE]: Term 1 election: Requested vote from peers 3991538cb6494302971d1b2153606935 (127.0.97.3:39651), b9ab54de55a14b6191ba59cffb758ad8 (127.0.97.1:39145)
I20260620 02:28:51.440858 5474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febbed8b24ae420fb879799ba8bdc5c1" candidate_uuid: "acdfa11624824eaaae1d393d87177825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3991538cb6494302971d1b2153606935"
I20260620 02:28:51.440927 5213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febbed8b24ae420fb879799ba8bdc5c1" candidate_uuid: "acdfa11624824eaaae1d393d87177825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9ab54de55a14b6191ba59cffb758ad8"
I20260620 02:28:51.441100 5474 raft_consensus.cc:3037] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:51.441187 5213 raft_consensus.cc:3037] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:51.443421 5474 raft_consensus.cc:2454] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acdfa11624824eaaae1d393d87177825 in term 1.
I20260620 02:28:51.443495 5213 raft_consensus.cc:2454] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acdfa11624824eaaae1d393d87177825 in term 1.
I20260620 02:28:51.443919 5279 leader_election.cc:302] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3991538cb6494302971d1b2153606935, acdfa11624824eaaae1d393d87177825; no voters:
I20260620 02:28:51.444279 5544 raft_consensus.cc:2781] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:51.444916 5544 raft_consensus.cc:686] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 1 LEADER]: Becoming Leader. State: Replica: acdfa11624824eaaae1d393d87177825, State: Running, Role: LEADER
I20260620 02:28:51.445484 5544 consensus_queue.cc:237] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } }
I20260620 02:28:51.450745 5071 catalog_manager.cc:5495] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 reported cstate change: term changed from 0 to 1, leader changed from <none> to acdfa11624824eaaae1d393d87177825 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "acdfa11624824eaaae1d393d87177825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acdfa11624824eaaae1d393d87177825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 } health_report { overall_health: UNKNOWN } } }
W20260620 02:28:51.464108 5390 tablet.cc:2369] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:28:51.464108 5520 tablet.cc:2369] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:51.497433 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.495175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.97.2:41465)
0620 02:28:51.495347 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.495352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.495371 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:28:51.495602 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.495606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.495688 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.495774 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.496689 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.496694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.496698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.496717 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.496751 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.496763 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.496866 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.497140 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.497195 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:28:51.506531 5474 raft_consensus.cc:1261] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 1 FOLLOWER]: Refusing update from remote peer acdfa11624824eaaae1d393d87177825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:51.506531 5213 raft_consensus.cc:1261] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 1 FOLLOWER]: Refusing update from remote peer acdfa11624824eaaae1d393d87177825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:28:51.507740 5548 consensus_queue.cc:1035] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9ab54de55a14b6191ba59cffb758ad8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:28:51.508139 5544 consensus_queue.cc:1035] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3991538cb6494302971d1b2153606935" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:28:51.509209 5259 tablet.cc:2369] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:28:51.539803 5070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:51.540215 5070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:28:51.544518 5070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 161b4ac0a1824e51a262dd68f30245a2: Sending DeleteTablet for 3 replicas of tablet febbed8b24ae420fb879799ba8bdc5c1
I20260620 02:28:51.545924 5454 tablet_service.cc:1514] Processing DeleteTablet for tablet febbed8b24ae420fb879799ba8bdc5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:51 UTC) from {username='slave'} at 127.0.0.1:43258
I20260620 02:28:51.545938 5324 tablet_service.cc:1514] Processing DeleteTablet for tablet febbed8b24ae420fb879799ba8bdc5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:51 UTC) from {username='slave'} at 127.0.0.1:42564
I20260620 02:28:51.546145 5193 tablet_service.cc:1514] Processing DeleteTablet for tablet febbed8b24ae420fb879799ba8bdc5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:28:51 UTC) from {username='slave'} at 127.0.0.1:48990
I20260620 02:28:51.551970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5131
I20260620 02:28:51.558948 5563 tablet_replica.cc:330] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: stopping tablet replica
I20260620 02:28:51.559659 5563 raft_consensus.cc:2229] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:51.560317 5563 raft_consensus.cc:2258] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:51.562918 5563 ts_tablet_manager.cc:1905] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:51.571138 5563 ts_tablet_manager.cc:1918] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:51.571410 5563 log.cc:1196] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/febbed8b24ae420fb879799ba8bdc5c1
I20260620 02:28:51.572059 5563 ts_tablet_manager.cc:1939] T febbed8b24ae420fb879799ba8bdc5c1 P b9ab54de55a14b6191ba59cffb758ad8: Deleting consensus metadata
I20260620 02:28:51.574249 5057 catalog_manager.cc:4841] TS b9ab54de55a14b6191ba59cffb758ad8 (127.0.97.1:39145): tablet febbed8b24ae420fb879799ba8bdc5c1 (table test-table [id=d54ccdfa90304f1eba119423eedd7983]) successfully deleted
I20260620 02:28:51.575501 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.554352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.97.1:39145)
0620 02:28:51.554602 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.554611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.572198 (+ 17587us) server_negotiation.cc:407] Connection header received
0620 02:28:51.572878 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.572888 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.573058 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.573186 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.574548 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.574558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.574566 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.574588 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.574637 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.574659 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.574791 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.575153 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.575218 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:28:51.576268 5253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:51.576774 5562 tablet_replica.cc:330] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: stopping tablet replica
I20260620 02:28:51.577281 5562 raft_consensus.cc:2229] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:28:51.577744 5562 raft_consensus.cc:2258] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:51.580531 5562 ts_tablet_manager.cc:1905] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:51.582329 5561 tablet_replica.cc:330] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: stopping tablet replica
I20260620 02:28:51.582973 5561 raft_consensus.cc:2229] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:28:51.583894 5561 raft_consensus.cc:2258] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:28:51.586361 5561 ts_tablet_manager.cc:1905] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:28:51.588047 5562 ts_tablet_manager.cc:1918] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:51.588274 5562 log.cc:1196] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/febbed8b24ae420fb879799ba8bdc5c1
I20260620 02:28:51.588876 5562 ts_tablet_manager.cc:1939] T febbed8b24ae420fb879799ba8bdc5c1 P 3991538cb6494302971d1b2153606935: Deleting consensus metadata
I20260620 02:28:51.590895 5058 catalog_manager.cc:4841] TS 3991538cb6494302971d1b2153606935 (127.0.97.3:39651): tablet febbed8b24ae420fb879799ba8bdc5c1 (table test-table [id=d54ccdfa90304f1eba119423eedd7983]) successfully deleted
I20260620 02:28:51.593211 5561 ts_tablet_manager.cc:1918] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:28:51.593541 5561 log.cc:1196] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/febbed8b24ae420fb879799ba8bdc5c1
I20260620 02:28:51.594255 5561 ts_tablet_manager.cc:1939] T febbed8b24ae420fb879799ba8bdc5c1 P acdfa11624824eaaae1d393d87177825: Deleting consensus metadata
I20260620 02:28:51.596673 5056 catalog_manager.cc:4841] TS acdfa11624824eaaae1d393d87177825 (127.0.97.2:41465): tablet febbed8b24ae420fb879799ba8bdc5c1 (table test-table [id=d54ccdfa90304f1eba119423eedd7983]) successfully deleted
I20260620 02:28:51.683547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5131
I20260620 02:28:51.707438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5262
I20260620 02:28:51.711547 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.708811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.0.97.2:41465)
0620 02:28:51.709046 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.709052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.709072 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:28:51.709152 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.709158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.709222 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.709321 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.710461 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.710466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.710471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.710489 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.710525 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.710539 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.710656 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.711196 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.711254 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:28:51.713732 5384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:51.825778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5262
I20260620 02:28:51.845197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5392
I20260620 02:28:51.848352 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:51.846081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.0.97.3:39651)
0620 02:28:51.846270 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:51.846274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:51.846471 (+ 197us) server_negotiation.cc:407] Connection header received
0620 02:28:51.846576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:51.846580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:51.846677 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:51.846771 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:51.847726 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:51.847731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:51.847734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:51.847749 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:51.847780 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:51.847792 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:51.847904 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:51.847961 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 02:28:51.848010 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:28:51.849058 5514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:51.938062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5392
I20260620 02:28:51.954931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5041
I20260620 02:28:51.956099 5102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:52.066968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5041
2026-06-20T02:28:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2870 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 02:28:52.109552 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:28:52.112247 388 webserver.cc:460] Webserver started at https://127.0.0.1:42967/ using document root <none> and password file <none>
I20260620 02:28:52.112646 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:28:52.113451 388 webserver.cc:460] Webserver started at http://127.0.0.1:44847/ using document root <none> and password file <none>
2026-06-20T02:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:52Z Disabled control of system clock
I20260620 02:28:52.145092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34519
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34519
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42967/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/testchainca.cert with env {}
W20260620 02:28:52.472581 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:52.472893 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:52.473099 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:52.480513 5574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:52.480639 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:52.480712 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:52.480800 5574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:52.480862 5574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:28:52.480980 5574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:28:52.481083 5574 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42967/jwks.json
I20260620 02:28:52.491487 5574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42967/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:52.492879 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:52.494242 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:52.506318 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:52.506377 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:52.506389 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:52.510643 5574 server_base.cc:1034] running on GCE node
I20260620 02:28:52.511479 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:52.512951 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:52.514240 5574 hybrid_clock.cc:648] HybridClock initialized: now 1781922532514174 us; error 69 us; skew 500 ppm
I20260620 02:28:52.514626 5574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:52.516937 5574 webserver.cc:460] Webserver started at http://127.0.97.62:43895/ using document root <none> and password file <none>
I20260620 02:28:52.517680 5574 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:52.517776 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:52.518051 5574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:52.521104 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "60eb4fdba29349059f7b5d76145b20cb"
format_stamp: "Formatted at 2026-06-20 02:28:52 on dist-test-slave-r706"
server_key: "acdaa50a23f0cb12ade73b2d753839d5"
server_key_iv: "54fa092c2901f07892396dc8ad40285f"
server_key_version: "encryptionkey@0"
I20260620 02:28:52.521911 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "60eb4fdba29349059f7b5d76145b20cb"
format_stamp: "Formatted at 2026-06-20 02:28:52 on dist-test-slave-r706"
server_key: "acdaa50a23f0cb12ade73b2d753839d5"
server_key_iv: "54fa092c2901f07892396dc8ad40285f"
server_key_version: "encryptionkey@0"
I20260620 02:28:52.527195 5574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:28:52.530833 5588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:52.532226 5574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:28:52.532389 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "60eb4fdba29349059f7b5d76145b20cb"
format_stamp: "Formatted at 2026-06-20 02:28:52 on dist-test-slave-r706"
server_key: "acdaa50a23f0cb12ade73b2d753839d5"
server_key_iv: "54fa092c2901f07892396dc8ad40285f"
server_key_version: "encryptionkey@0"
I20260620 02:28:52.532589 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:52.546638 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:52.547470 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:52.547729 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:52.566814 5574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34519
I20260620 02:28:52.566828 5639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34519 every 8 connection(s)
I20260620 02:28:52.568274 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:52.572165 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:52.575163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5574
I20260620 02:28:52.575405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:52.576212 388 external_mini_cluster.cc:1428] Setting key 86f08f2009dae13887cd11075f1213ff
I20260620 02:28:52.585426 5640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb: Bootstrap starting.
I20260620 02:28:52.589484 5640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:52.590646 5640 log.cc:824] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:52.590827 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:52.578245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.0.97.62:34519)
0620 02:28:52.579327 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:52.579338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:52.579385 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:28:52.581189 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:52.581215 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:52.581736 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:52.582112 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:52.582803 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:52.584297 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:52.586237 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:52.586910 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:52.587081 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:52.588396 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:52.588404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:52.588417 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:52.588966 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:52.589148 (+ 182us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:52.589171 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:52.589262 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:52.589520 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 02:28:52.589845 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":412,"threads_started":1}
I20260620 02:28:52.593577 5640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb: No bootstrap required, opened a new log
I20260620 02:28:52.594604 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:28:52.601933 5640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eb4fdba29349059f7b5d76145b20cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260620 02:28:52.602329 5640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:52.602460 5640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60eb4fdba29349059f7b5d76145b20cb, State: Initialized, Role: FOLLOWER
I20260620 02:28:52.603247 5640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eb4fdba29349059f7b5d76145b20cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260620 02:28:52.603615 5640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:52.603773 5640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:52.603984 5640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:52.606269 5640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eb4fdba29349059f7b5d76145b20cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260620 02:28:52.606812 5640 leader_election.cc:302] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60eb4fdba29349059f7b5d76145b20cb; no voters:
I20260620 02:28:52.607511 5640 leader_election.cc:288] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:52.607600 5645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:52.608485 5645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [term 1 LEADER]: Becoming Leader. State: Replica: 60eb4fdba29349059f7b5d76145b20cb, State: Running, Role: LEADER
I20260620 02:28:52.609138 5645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eb4fdba29349059f7b5d76145b20cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260620 02:28:52.609800 5640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:52.611776 5647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60eb4fdba29349059f7b5d76145b20cb. Latest consensus state: current_term: 1 leader_uuid: "60eb4fdba29349059f7b5d76145b20cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eb4fdba29349059f7b5d76145b20cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } } }
I20260620 02:28:52.612169 5647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:52.613212 5646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60eb4fdba29349059f7b5d76145b20cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eb4fdba29349059f7b5d76145b20cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } } }
I20260620 02:28:52.614048 5646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:52.617709 5650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:52.622979 5650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:52.632707 5650 catalog_manager.cc:1347] Generated new cluster ID: 2a5f818794ff4d7aa17a78c8fc76f9af
I20260620 02:28:52.632846 5650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:52.648551 5650 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:52.650025 5650 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:52.659762 5650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60eb4fdba29349059f7b5d76145b20cb: Generated new TSK 0
I20260620 02:28:52.660734 5650 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:28:52.699007 5671 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 02:28:52.699221 5671 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:28:52.700533 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:52.677572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.97.62:34519)
0620 02:28:52.677784 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:52.677792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:52.678284 (+ 492us) server_negotiation.cc:407] Connection header received
0620 02:28:52.678621 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:52.678627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:52.678737 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:52.678926 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:28:52.679565 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:52.680722 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:52.682574 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:52.683297 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:52.683443 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:52.684048 (+ 605us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:28:52.699740 (+ 15692us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:28:52.700198 (+ 458us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:28:52.700665 5643 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 02:28:52.702165 5670 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:52.677409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.0.1:40602)
0620 02:28:52.678170 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:52.678186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:52.678343 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:52.678985 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:52.678991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:52.679005 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:28:52.679394 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:52.679405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:52.680895 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:52.680900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:52.682398 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:52.682414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:52.683435 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:52.683511 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:52.683655 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:52.683723 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:28:52.700135 (+ 16412us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:28:52.700546 (+ 411us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34519: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":412,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:34519: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 02:28:52.704116 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5574
I20260620 02:28:52.705487 5635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:52.801116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5574
2026-06-20T02:28:52Z chronyd exiting
I20260620 02:28:52.852665 388 test_util.cc:175] -----------------------------------------------
I20260620 02:28:52.852854 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922485472743-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 02:28:52.857100 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:28:52.859043 388 webserver.cc:460] Webserver started at https://127.0.0.1:43121/ using document root <none> and password file <none>
I20260620 02:28:52.859311 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:28:52.860008 388 webserver.cc:460] Webserver started at http://127.0.0.1:44541/ using document root <none> and password file <none>
2026-06-20T02:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:52Z Disabled control of system clock
I20260620 02:28:52.890239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39903
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39903
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43121/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/testchainca.cert with env {}
W20260620 02:28:53.235167 5678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:53.235558 5678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:53.235741 5678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:53.243482 5678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:53.243673 5678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:53.243788 5678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:53.243916 5678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:53.244006 5678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:28:53.244115 5678 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:28:53.244213 5678 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43121/jwks.json
I20260620 02:28:53.255100 5678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43121/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:53.256547 5678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:53.258013 5678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:53.271001 5684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:53.270953 5683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:53.271471 5686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:53.272264 5678 server_base.cc:1034] running on GCE node
I20260620 02:28:53.273314 5678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:53.274883 5678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:53.276111 5678 hybrid_clock.cc:648] HybridClock initialized: now 1781922533276063 us; error 36 us; skew 500 ppm
I20260620 02:28:53.276508 5678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:53.278990 5678 webserver.cc:460] Webserver started at http://127.0.97.62:39223/ using document root <none> and password file <none>
I20260620 02:28:53.279743 5678 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:53.279873 5678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:53.280246 5678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:53.283428 5678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "29f7e77ea3eb41608f2213fbe81019f1"
format_stamp: "Formatted at 2026-06-20 02:28:53 on dist-test-slave-r706"
server_key: "5795f5d131874ab020193097dcf2c894"
server_key_iv: "72c16bddc2efceb5002f107ecf193e17"
server_key_version: "encryptionkey@0"
I20260620 02:28:53.284379 5678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "29f7e77ea3eb41608f2213fbe81019f1"
format_stamp: "Formatted at 2026-06-20 02:28:53 on dist-test-slave-r706"
server_key: "5795f5d131874ab020193097dcf2c894"
server_key_iv: "72c16bddc2efceb5002f107ecf193e17"
server_key_version: "encryptionkey@0"
I20260620 02:28:53.289968 5678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:28:53.293483 5692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:53.294940 5678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:28:53.295168 5678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "29f7e77ea3eb41608f2213fbe81019f1"
format_stamp: "Formatted at 2026-06-20 02:28:53 on dist-test-slave-r706"
server_key: "5795f5d131874ab020193097dcf2c894"
server_key_iv: "72c16bddc2efceb5002f107ecf193e17"
server_key_version: "encryptionkey@0"
I20260620 02:28:53.295433 5678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:53.309958 5678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:53.310799 5678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:53.311103 5678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:53.330147 5678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39903
I20260620 02:28:53.330170 5743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39903 every 8 connection(s)
I20260620 02:28:53.331583 5678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:53.332427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5678
I20260620 02:28:53.332733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:53.333498 388 external_mini_cluster.cc:1428] Setting key 7dbfdffb1bad609a0a331abdf6d8e2be
I20260620 02:28:53.336220 5744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:53.349478 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:53.335483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.97.62:39903)
0620 02:28:53.336576 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:53.336591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:53.336642 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:28:53.338973 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:53.339000 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:53.339616 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:53.340073 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:53.340605 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:53.342732 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:53.344055 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:53.345096 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:53.345274 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:53.346513 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:53.346524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:53.346543 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:53.347338 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:53.347445 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:53.347471 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:53.347595 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:53.347813 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 02:28:53.348175 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":369,"threads_started":1}
I20260620 02:28:53.350477 5744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1: Bootstrap starting.
I20260620 02:28:53.353717 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:28:53.354442 5744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:53.355677 5744 log.cc:824] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:53.358731 5744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1: No bootstrap required, opened a new log
I20260620 02:28:53.366163 5744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f7e77ea3eb41608f2213fbe81019f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39903 } }
I20260620 02:28:53.366528 5744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:53.366647 5744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f7e77ea3eb41608f2213fbe81019f1, State: Initialized, Role: FOLLOWER
I20260620 02:28:53.367441 5744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f7e77ea3eb41608f2213fbe81019f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39903 } }
I20260620 02:28:53.367691 5744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:53.367882 5744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:53.368052 5744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:53.370529 5744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f7e77ea3eb41608f2213fbe81019f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39903 } }
I20260620 02:28:53.371078 5744 leader_election.cc:302] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f7e77ea3eb41608f2213fbe81019f1; no voters:
I20260620 02:28:53.371832 5744 leader_election.cc:288] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:53.372020 5749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:53.372731 5749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [term 1 LEADER]: Becoming Leader. State: Replica: 29f7e77ea3eb41608f2213fbe81019f1, State: Running, Role: LEADER
I20260620 02:28:53.373319 5749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f7e77ea3eb41608f2213fbe81019f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39903 } }
I20260620 02:28:53.374065 5744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:53.376386 5750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f7e77ea3eb41608f2213fbe81019f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f7e77ea3eb41608f2213fbe81019f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39903 } } }
I20260620 02:28:53.376405 5751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f7e77ea3eb41608f2213fbe81019f1. Latest consensus state: current_term: 1 leader_uuid: "29f7e77ea3eb41608f2213fbe81019f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f7e77ea3eb41608f2213fbe81019f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39903 } } }
I20260620 02:28:53.376899 5750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:53.376905 5751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:53.377864 5756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:53.383258 5756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:53.393496 5756 catalog_manager.cc:1347] Generated new cluster ID: 3a5a344b120c4c3c8b2315a42093e06e
I20260620 02:28:53.393644 5756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:53.402829 5756 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:53.404147 5756 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:53.415759 5756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f7e77ea3eb41608f2213fbe81019f1: Generated new TSK 0
I20260620 02:28:53.416628 5756 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:28:53.459729 5774 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 02:28:53.459878 5774 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:28:53.460956 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:53.432414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.0.97.62:39903)
0620 02:28:53.432648 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:53.432655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:53.433076 (+ 421us) server_negotiation.cc:407] Connection header received
0620 02:28:53.433293 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:53.433297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:53.433411 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:53.433553 (+ 142us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:28:53.434173 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:53.434912 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:53.437153 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:53.437674 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:53.437785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:53.438514 (+ 729us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:28:53.460316 (+ 21802us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:28:53.460697 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":98}
W20260620 02:28:53.461138 5747 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 02:28:53.461361 5773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:53.432174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39903 (local address 127.0.0.1:45828)
0620 02:28:53.432888 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:53.432912 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:53.433108 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:53.433608 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:53.433613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:53.433629 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:28:53.433987 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:53.434001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:53.435070 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:53.435075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:53.436879 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:53.436897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:53.437820 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:53.437896 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:53.438128 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:53.438180 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:28:53.460605 (+ 22425us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:28:53.460916 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":358,"threads_started":1}
I20260620 02:28:53.462873 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5678
I20260620 02:28:53.464118 5739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:53.558573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5678
2026-06-20T02:28:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (757 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 02:28:53.614802 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:28:53.616710 388 webserver.cc:460] Webserver started at https://127.0.0.1:41873/ using document root <none> and password file <none>
I20260620 02:28:53.616988 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:28:53.617697 388 webserver.cc:460] Webserver started at http://127.0.0.1:40337/ using document root <none> and password file <none>
2026-06-20T02:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:53Z Disabled control of system clock
I20260620 02:28:53.648277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40971
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:28:53.980165 5781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:53.980492 5781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:53.980584 5781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:53.988835 5781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:53.988968 5781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:53.989069 5781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:53.989156 5781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:53.989197 5781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:28:53.989254 5781 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:28:53.989288 5781 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41873/jwks.json
I20260620 02:28:53.998801 5781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:54.000062 5781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:54.001863 5781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:54.013861 5789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:54.013928 5787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:54.014128 5786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:54.014873 5781 server_base.cc:1034] running on GCE node
I20260620 02:28:54.015719 5781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:54.017274 5781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:54.018517 5781 hybrid_clock.cc:648] HybridClock initialized: now 1781922534018481 us; error 55 us; skew 500 ppm
I20260620 02:28:54.018873 5781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:54.021627 5781 webserver.cc:460] Webserver started at http://127.0.97.62:40073/ using document root <none> and password file <none>
I20260620 02:28:54.022337 5781 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:54.022466 5781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:54.022837 5781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:54.025810 5781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "0b3823f825354166920d44f23eee64b4"
format_stamp: "Formatted at 2026-06-20 02:28:54 on dist-test-slave-r706"
server_key: "1e791207d266fdfd7c6b7ba5f9a48e67"
server_key_iv: "e933187b1dc575d63d750983d0d5a1b3"
server_key_version: "encryptionkey@0"
I20260620 02:28:54.026609 5781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b3823f825354166920d44f23eee64b4"
format_stamp: "Formatted at 2026-06-20 02:28:54 on dist-test-slave-r706"
server_key: "1e791207d266fdfd7c6b7ba5f9a48e67"
server_key_iv: "e933187b1dc575d63d750983d0d5a1b3"
server_key_version: "encryptionkey@0"
I20260620 02:28:54.031687 5781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:28:54.035418 5795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:54.036733 5781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:54.036953 5781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "0b3823f825354166920d44f23eee64b4"
format_stamp: "Formatted at 2026-06-20 02:28:54 on dist-test-slave-r706"
server_key: "1e791207d266fdfd7c6b7ba5f9a48e67"
server_key_iv: "e933187b1dc575d63d750983d0d5a1b3"
server_key_version: "encryptionkey@0"
I20260620 02:28:54.037245 5781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:54.052137 5781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:54.053089 5781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:54.053404 5781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:54.072193 5781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40971
I20260620 02:28:54.072201 5846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40971 every 8 connection(s)
I20260620 02:28:54.073741 5781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:54.077540 5847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:54.079777 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5781
I20260620 02:28:54.080025 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:54.080693 388 external_mini_cluster.cc:1428] Setting key 3453382df84cd7d75641518fd38ea44d
I20260620 02:28:54.089576 5847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4: Bootstrap starting.
I20260620 02:28:54.093413 5847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:54.094523 5847 log.cc:824] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:54.095038 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:54.082814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.0.97.62:40971)
0620 02:28:54.083834 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:54.083844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:54.083887 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:28:54.085627 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:54.085650 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:54.086146 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:54.086505 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:54.087199 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.088500 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.090311 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.090992 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.091144 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:54.092441 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:54.092450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:54.092463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:54.093056 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:54.093162 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:54.093189 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:54.093307 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:54.093625 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:28:54.094003 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":407,"threads_started":1}
I20260620 02:28:54.097702 5847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4: No bootstrap required, opened a new log
I20260620 02:28:54.098950 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:28:54.105407 5847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3823f825354166920d44f23eee64b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 02:28:54.105832 5847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:54.105957 5847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3823f825354166920d44f23eee64b4, State: Initialized, Role: FOLLOWER
I20260620 02:28:54.106608 5847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3823f825354166920d44f23eee64b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 02:28:54.106850 5847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:54.107023 5847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:54.107203 5847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:54.109503 5847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3823f825354166920d44f23eee64b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 02:28:54.110023 5847 leader_election.cc:302] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b3823f825354166920d44f23eee64b4; no voters:
I20260620 02:28:54.110608 5847 leader_election.cc:288] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:54.110811 5852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:54.112011 5852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b3823f825354166920d44f23eee64b4, State: Running, Role: LEADER
I20260620 02:28:54.112534 5852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3823f825354166920d44f23eee64b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } }
I20260620 02:28:54.112848 5847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:54.115497 5854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3823f825354166920d44f23eee64b4. Latest consensus state: current_term: 1 leader_uuid: "0b3823f825354166920d44f23eee64b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3823f825354166920d44f23eee64b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } } }
I20260620 02:28:54.115953 5854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:54.116561 5853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b3823f825354166920d44f23eee64b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3823f825354166920d44f23eee64b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40971 } } }
I20260620 02:28:54.116899 5853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:54.116948 5861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:54.123704 5861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:54.133713 5861 catalog_manager.cc:1347] Generated new cluster ID: 5cdcdd27389f4a3fa6927e7cc0d9c264
I20260620 02:28:54.133839 5861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:54.142284 5861 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:54.144060 5861 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:54.154995 5861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b3823f825354166920d44f23eee64b4: Generated new TSK 0
I20260620 02:28:54.155870 5861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:28:54.180562 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:54.171974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.0.97.62:40971)
0620 02:28:54.172184 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:54.172191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:54.172691 (+ 500us) server_negotiation.cc:407] Connection header received
0620 02:28:54.172904 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:54.172910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:54.173040 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:54.173218 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:28:54.173662 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.174786 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.176126 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.176793 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.176917 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:54.177505 (+ 588us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:28:54.179910 (+ 2405us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:28:54.180232 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 02:28:54.180701 5850 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 02:28:54.182382 5876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:54.171905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40971 (local address 127.0.0.1:59320)
0620 02:28:54.172585 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:54.172600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:54.172719 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:54.173224 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:54.173227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:54.173237 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:28:54.173508 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:54.173516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.174928 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.174931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:54.175982 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:54.175992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.176929 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.176986 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:54.177189 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:54.177223 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:28:54.180109 (+ 2886us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:28:54.181962 (+ 1853us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":1537,"thread_start_us":384,"threads_started":1}
I20260620 02:28:54.184451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5781
I20260620 02:28:54.185635 5842 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 02:28:54.192348 5877 webserver.cc:533] Webserver: sslize failed:
E20260620 02:28:54.192488 5877 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:28:54.275111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5781
2026-06-20T02:28:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (717 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T02:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:54Z Disabled control of system clock
I20260620 02:28:54.362061 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34919
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34919
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/testchainca.cert with env {}
W20260620 02:28:54.689282 5882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:54.689594 5882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:54.689683 5882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:54.696913 5882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:54.697098 5882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:54.697168 5882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:54.697221 5882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:54.697266 5882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:28:54.697322 5882 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:28:54.697362 5882 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 02:28:54.707399 5882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:54.708698 5882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:54.710011 5882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:54.722426 5888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:54.722563 5890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:54.722404 5887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:54.723212 5882 server_base.cc:1034] running on GCE node
I20260620 02:28:54.724089 5882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:54.725636 5882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:54.726888 5882 hybrid_clock.cc:648] HybridClock initialized: now 1781922534726844 us; error 41 us; skew 500 ppm
I20260620 02:28:54.727252 5882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:28:54.734295 5882 webserver.cc:460] Webserver started at http://127.0.97.62:33271/ using document root <none> and password file <none>
I20260620 02:28:54.735158 5882 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:54.735330 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:54.735677 5882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:54.738811 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "5e45b4aabe0648dc8955405dfa9086dc"
format_stamp: "Formatted at 2026-06-20 02:28:54 on dist-test-slave-r706"
server_key: "2ba5a34fa2c2ef55f4ffe1208d6c10d4"
server_key_iv: "73a217855bc6ef5e7ef537f495b9de2b"
server_key_version: "encryptionkey@0"
I20260620 02:28:54.739717 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e45b4aabe0648dc8955405dfa9086dc"
format_stamp: "Formatted at 2026-06-20 02:28:54 on dist-test-slave-r706"
server_key: "2ba5a34fa2c2ef55f4ffe1208d6c10d4"
server_key_iv: "73a217855bc6ef5e7ef537f495b9de2b"
server_key_version: "encryptionkey@0"
I20260620 02:28:54.745260 5882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:28:54.749168 5896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:54.750617 5882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:54.750823 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "5e45b4aabe0648dc8955405dfa9086dc"
format_stamp: "Formatted at 2026-06-20 02:28:54 on dist-test-slave-r706"
server_key: "2ba5a34fa2c2ef55f4ffe1208d6c10d4"
server_key_iv: "73a217855bc6ef5e7ef537f495b9de2b"
server_key_version: "encryptionkey@0"
I20260620 02:28:54.751068 5882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:54.767025 5882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:54.767900 5882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:54.768210 5882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:54.787096 5882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34919
I20260620 02:28:54.787115 5947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34919 every 8 connection(s)
I20260620 02:28:54.788555 5882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:54.792256 5948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:54.793210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 5882
I20260620 02:28:54.793428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:54.794288 388 external_mini_cluster.cc:1428] Setting key 018f896588e8c57fded5cb0aa7463afe
I20260620 02:28:54.805621 5948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc: Bootstrap starting.
I20260620 02:28:54.807801 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:54.796309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.0.97.62:34919)
0620 02:28:54.797323 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:54.797334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:54.797376 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:28:54.798972 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:54.798994 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:54.799471 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:54.799882 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:54.800448 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.801811 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.803103 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.803827 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.804038 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:54.805338 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:54.805346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:54.805361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:28:54.806012 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:54.806097 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:28:54.806123 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:54.806260 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:54.806502 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 02:28:54.806840 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":314,"threads_started":1}
I20260620 02:28:54.809226 5948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:54.810362 5948 log.cc:824] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:54.811875 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:28:54.813834 5948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc: No bootstrap required, opened a new log
I20260620 02:28:54.822084 5948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e45b4aabe0648dc8955405dfa9086dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34919 } }
I20260620 02:28:54.822481 5948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:54.822605 5948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e45b4aabe0648dc8955405dfa9086dc, State: Initialized, Role: FOLLOWER
I20260620 02:28:54.823187 5948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e45b4aabe0648dc8955405dfa9086dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34919 } }
I20260620 02:28:54.823393 5948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:54.823566 5948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:54.823745 5948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:54.826027 5948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e45b4aabe0648dc8955405dfa9086dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34919 } }
I20260620 02:28:54.826539 5948 leader_election.cc:302] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e45b4aabe0648dc8955405dfa9086dc; no voters:
I20260620 02:28:54.827144 5948 leader_election.cc:288] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:54.827337 5953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:54.828291 5953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [term 1 LEADER]: Becoming Leader. State: Replica: 5e45b4aabe0648dc8955405dfa9086dc, State: Running, Role: LEADER
I20260620 02:28:54.828787 5953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e45b4aabe0648dc8955405dfa9086dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34919 } }
I20260620 02:28:54.829392 5948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:54.831882 5954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e45b4aabe0648dc8955405dfa9086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e45b4aabe0648dc8955405dfa9086dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34919 } } }
I20260620 02:28:54.832363 5954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:54.832546 5955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e45b4aabe0648dc8955405dfa9086dc. Latest consensus state: current_term: 1 leader_uuid: "5e45b4aabe0648dc8955405dfa9086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e45b4aabe0648dc8955405dfa9086dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34919 } } }
I20260620 02:28:54.832991 5955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:54.839591 5960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:54.844733 5960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:54.854292 5960 catalog_manager.cc:1347] Generated new cluster ID: 04852913f5234c009cf4f6803a489533
I20260620 02:28:54.854439 5960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:54.872651 5960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:54.874500 5960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:54.886267 5960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e45b4aabe0648dc8955405dfa9086dc: Generated new TSK 0
I20260620 02:28:54.887421 5960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:28:55.423689 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:54.907568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.0.97.62:34919)
0620 02:28:54.907814 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:54.907824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:54.908268 (+ 444us) server_negotiation.cc:407] Connection header received
0620 02:28:54.908494 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:54.908498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:54.908573 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:54.908711 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:28:54.909301 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.910058 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.912182 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.912612 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.912704 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:54.913369 (+ 665us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:28:55.423100 (+509731us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:28:55.423469 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44748: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:28:55.423794 5951 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44748: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 02:28:55.424068 5977 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:54.907510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34919 (local address 127.0.0.1:44748)
0620 02:28:54.908146 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:54.908168 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:54.908328 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:54.908771 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:54.908776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:54.908790 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:28:54.909145 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:54.909158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.910218 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.910223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:54.912024 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:54.912040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:54.912737 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:54.912797 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:54.913056 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:54.913109 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:28:55.423418 (+510309us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:28:55.423656 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34919: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":337,"threads_started":1}
I20260620 02:28:55.425704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 5882
I20260620 02:28:55.426803 5943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:28:55.529876 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 5882
2026-06-20T02:28:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1235 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (417 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (418 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:28:56 dist-test-slave-r706 krb5kdc[5992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:28:56 dist-test-slave-r706 krb5kdc[5992](info): set up 2 sockets
Jun 20 02:28:56 dist-test-slave-r706 krb5kdc[5992](info): commencing operation
krb5kdc: starting...
W20260620 02:28:57.469910 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:28:57 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:28:57.700714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43997
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:58.042191 6005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:58.042626 6005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:58.042837 6005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:58.051765 6005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:28:58.051990 6005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:58.052119 6005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:58.052228 6005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:28:58.052316 6005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:28:58.062819 6005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:58.064293 6005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:58.065723 6005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:58.077875 6010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:58.078018 6011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:58.078223 6013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:58.079190 6005 server_base.cc:1034] running on GCE node
I20260620 02:28:58.080049 6005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:58.081677 6005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:58.082917 6005 hybrid_clock.cc:648] HybridClock initialized: now 1781922538082840 us; error 65 us; skew 500 ppm
I20260620 02:28:58.083272 6005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:58 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:58.086505 6005 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:28:58.087967 6005 webserver.cc:460] Webserver started at http://127.0.97.62:41633/ using document root <none> and password file <none>
I20260620 02:28:58.088636 6005 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:58.088743 6005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:58.089049 6005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:58.092234 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "6d2ffebd18b444eaa112e25ef3696370"
format_stamp: "Formatted at 2026-06-20 02:28:58 on dist-test-slave-r706"
server_key: "769e234314fc08f27026a3a26912770a"
server_key_iv: "963d1b66b884d4565da9af190ca56767"
server_key_version: "encryptionkey@0"
I20260620 02:28:58.093128 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d2ffebd18b444eaa112e25ef3696370"
format_stamp: "Formatted at 2026-06-20 02:28:58 on dist-test-slave-r706"
server_key: "769e234314fc08f27026a3a26912770a"
server_key_iv: "963d1b66b884d4565da9af190ca56767"
server_key_version: "encryptionkey@0"
I20260620 02:28:58.098917 6005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 02:28:58.102790 6020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:58.104348 6005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:28:58.104557 6005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "6d2ffebd18b444eaa112e25ef3696370"
format_stamp: "Formatted at 2026-06-20 02:28:58 on dist-test-slave-r706"
server_key: "769e234314fc08f27026a3a26912770a"
server_key_iv: "963d1b66b884d4565da9af190ca56767"
server_key_version: "encryptionkey@0"
I20260620 02:28:58.104769 6005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:58.120434 6005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:58.125188 6005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:58.125499 6005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:58.145355 6005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43997
I20260620 02:28:58.145406 6071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43997 every 8 connection(s)
I20260620 02:28:58.147079 6005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:28:58.151832 6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:28:58.153497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6005
I20260620 02:28:58.153731 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:28:58.154386 388 external_mini_cluster.cc:1428] Setting key 5cb409693ed622d85a0c898843385d20
I20260620 02:28:58.167071 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370: Bootstrap starting.
Jun 20 02:28:58 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:58.170421 6072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370: Neither blocks nor log segments found. Creating new log.
I20260620 02:28:58.172042 6072 log.cc:824] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370: Log is configured to *not* fsync() on all Append() calls
I20260620 02:28:58.175978 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370: No bootstrap required, opened a new log
I20260620 02:28:58.180773 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:58.156482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.97.62:43997)
0620 02:28:58.157358 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:58.157376 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:58.157451 (+ 75us) server_negotiation.cc:407] Connection header received
0620 02:28:58.159290 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:58.159321 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:58.159884 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:58.160322 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:58.161780 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:58.163879 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:58.165498 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:58.166602 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:58.166803 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:58.171115 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:58.171128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:58.171150 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:58.171200 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:58.175606 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:58.176520 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:58.176532 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:58.176546 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:58.176765 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:58.177252 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:58.177258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:58.177262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:58.178073 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:58.178406 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:58.178780 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 02:28:58.179293 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":338,"threads_started":1}
I20260620 02:28:58.186055 6072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2ffebd18b444eaa112e25ef3696370" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43997 } }
I20260620 02:28:58.186424 6072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:28:58.186506 6072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2ffebd18b444eaa112e25ef3696370, State: Initialized, Role: FOLLOWER
I20260620 02:28:58.187124 6072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2ffebd18b444eaa112e25ef3696370" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43997 } }
I20260620 02:28:58.187386 6072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:28:58.187538 6072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:28:58.187745 6072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:28:58.190340 6072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2ffebd18b444eaa112e25ef3696370" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43997 } }
I20260620 02:28:58.190965 6072 leader_election.cc:302] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d2ffebd18b444eaa112e25ef3696370; no voters:
I20260620 02:28:58.191725 6072 leader_election.cc:288] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:28:58.191852 6077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:28:58.192724 6077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [term 1 LEADER]: Becoming Leader. State: Replica: 6d2ffebd18b444eaa112e25ef3696370, State: Running, Role: LEADER
I20260620 02:28:58.193491 6077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2ffebd18b444eaa112e25ef3696370" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43997 } }
I20260620 02:28:58.194972 6072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:28:58.196604 6078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d2ffebd18b444eaa112e25ef3696370. Latest consensus state: current_term: 1 leader_uuid: "6d2ffebd18b444eaa112e25ef3696370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2ffebd18b444eaa112e25ef3696370" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43997 } } }
I20260620 02:28:58.196671 6079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d2ffebd18b444eaa112e25ef3696370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2ffebd18b444eaa112e25ef3696370" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43997 } } }
I20260620 02:28:58.197067 6078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:58.197131 6079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370 [sys.catalog]: This master's current role is: LEADER
I20260620 02:28:58.200186 6082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:28:58.205595 6082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:28:58.215687 6082 catalog_manager.cc:1347] Generated new cluster ID: 8740fb19391a4b7abe9d8acdbf5788d7
I20260620 02:28:58.215906 6082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:28:58.245476 6082 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:28:58.247121 6082 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:28:58.257763 6082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d2ffebd18b444eaa112e25ef3696370: Generated new TSK 0
I20260620 02:28:58.258711 6082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:28:58.329274 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43997
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:58.674840 6098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:58.675187 6098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:58.675369 6098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:58.682977 6098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:58.683095 6098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:58.683295 6098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:28:58.693423 6098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:58.694813 6098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:58.696250 6098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:58.710809 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:58.710884 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:58.710892 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:58.711704 6098 server_base.cc:1034] running on GCE node
I20260620 02:28:58.712296 6098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:58.713581 6098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:58.714823 6098 hybrid_clock.cc:648] HybridClock initialized: now 1781922538714770 us; error 34 us; skew 500 ppm
I20260620 02:28:58.715193 6098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:58 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:58.718674 6098 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:28:58.720373 6098 webserver.cc:460] Webserver started at http://127.0.97.1:39893/ using document root <none> and password file <none>
I20260620 02:28:58.721213 6098 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:58.721355 6098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:58.721709 6098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:58.724838 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "1fdfebfd50e84174bdfadd7937e5d7bb"
format_stamp: "Formatted at 2026-06-20 02:28:58 on dist-test-slave-r706"
server_key: "dc0ac6e2430ae16cc8785fe02542f236"
server_key_iv: "61d71455bc6898b3aad047b02c8cc72e"
server_key_version: "encryptionkey@0"
I20260620 02:28:58.725773 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1fdfebfd50e84174bdfadd7937e5d7bb"
format_stamp: "Formatted at 2026-06-20 02:28:58 on dist-test-slave-r706"
server_key: "dc0ac6e2430ae16cc8785fe02542f236"
server_key_iv: "61d71455bc6898b3aad047b02c8cc72e"
server_key_version: "encryptionkey@0"
I20260620 02:28:58.731488 6098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:58.735679 6113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:58.737263 6098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:28:58.737514 6098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "1fdfebfd50e84174bdfadd7937e5d7bb"
format_stamp: "Formatted at 2026-06-20 02:28:58 on dist-test-slave-r706"
server_key: "dc0ac6e2430ae16cc8785fe02542f236"
server_key_iv: "61d71455bc6898b3aad047b02c8cc72e"
server_key_version: "encryptionkey@0"
I20260620 02:28:58.737766 6098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:58.760022 6098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:58.764312 6098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:58.764626 6098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:58.765717 6098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:58.767102 6098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:58.767215 6098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:58.767377 6098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:58.767452 6098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:58.802649 6098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33937
I20260620 02:28:58.802709 6225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33937 every 8 connection(s)
I20260620 02:28:58.804105 6098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:28:58.814548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6098
I20260620 02:28:58.814769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:28:58.815616 388 external_mini_cluster.cc:1428] Setting key f620ecc86920cb46e25275ca0f68d81c
Jun 20 02:28:58 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:58.826785 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:58.807007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48455 (local address 127.0.97.62:43997)
0620 02:28:58.807519 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:58.807526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:58.808640 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 02:28:58.810586 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:58.810591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:58.810660 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:58.810763 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:58.812334 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:58.813219 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:58.815572 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:58.816045 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:58.816155 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:58.820779 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:58.820784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:58.820788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:58.820821 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:58.823485 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:58.824296 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:58.824303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:58.824307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:58.824437 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:58.824895 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:58.824902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:58.824905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:58.825245 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:58.825404 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:58.826161 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 02:28:58.826397 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260620 02:28:58.827838 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:58.807606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43997 (local address 127.0.97.1:48455)
0620 02:28:58.808464 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:58.808519 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:58.810298 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:58.810999 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:58.811007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:58.811545 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:58.812161 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:58.812185 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:58.813402 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:58.813406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:58.815391 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:58.815411 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:58.816299 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:58.816306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:58.816535 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:58.817349 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:58.817381 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:58.820469 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:58.823654 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:58.823664 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:58.823685 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:58.824164 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:58.824563 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:58.824568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:58.824572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:58.824775 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:58.825376 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:58.825384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:58.825773 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 02:28:58.826052 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 02:28:58.826637 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":383,"threads_started":1}
I20260620 02:28:58.829526 6226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43997
I20260620 02:28:58.829886 6226 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:58.830842 6226 heartbeater.cc:511] Master 127.0.97.62:43997 requested a full tablet report, sending...
I20260620 02:28:58.833310 6037 ts_manager.cc:194] Registered new tserver with Master: 1fdfebfd50e84174bdfadd7937e5d7bb (127.0.97.1:33937)
I20260620 02:28:58.834745 6037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48455
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:28:58.869386 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43997
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:59.213330 6232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:59.213649 6232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:59.213771 6232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:59.221354 6232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:59.221490 6232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:59.221608 6232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:28:59.231909 6232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:59.233389 6232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:59.234737 6232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:59.249423 6238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:59.249518 6240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:59.249500 6237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:59.250916 6232 server_base.cc:1034] running on GCE node
I20260620 02:28:59.251652 6232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:59.252694 6232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:59.253942 6232 hybrid_clock.cc:648] HybridClock initialized: now 1781922539253863 us; error 77 us; skew 500 ppm
I20260620 02:28:59.254246 6232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.257496 6232 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:28:59.259167 6232 webserver.cc:460] Webserver started at http://127.0.97.2:44493/ using document root <none> and password file <none>
I20260620 02:28:59.259833 6232 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:59.259938 6232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:59.260203 6232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:59.263312 6232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "9a4308ecc8014b37bbef68a5aee04df5"
format_stamp: "Formatted at 2026-06-20 02:28:59 on dist-test-slave-r706"
server_key: "d532c70e33eb57cca927ece6b58cb49f"
server_key_iv: "525f1ecfb44c42b6f22ff6f76949cb43"
server_key_version: "encryptionkey@0"
I20260620 02:28:59.264073 6232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9a4308ecc8014b37bbef68a5aee04df5"
format_stamp: "Formatted at 2026-06-20 02:28:59 on dist-test-slave-r706"
server_key: "d532c70e33eb57cca927ece6b58cb49f"
server_key_iv: "525f1ecfb44c42b6f22ff6f76949cb43"
server_key_version: "encryptionkey@0"
I20260620 02:28:59.269582 6232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:28:59.273365 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:59.274922 6232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:28:59.275136 6232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "9a4308ecc8014b37bbef68a5aee04df5"
format_stamp: "Formatted at 2026-06-20 02:28:59 on dist-test-slave-r706"
server_key: "d532c70e33eb57cca927ece6b58cb49f"
server_key_iv: "525f1ecfb44c42b6f22ff6f76949cb43"
server_key_version: "encryptionkey@0"
I20260620 02:28:59.275352 6232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:59.308627 6232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:59.313197 6232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:59.313501 6232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:59.314530 6232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:59.316009 6232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:59.316085 6232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:59.316226 6232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:59.316278 6232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:59.354086 6232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38237
I20260620 02:28:59.354139 6359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38237 every 8 connection(s)
I20260620 02:28:59.355435 6232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:28:59.363435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6232
I20260620 02:28:59.363628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:28:59.364137 388 external_mini_cluster.cc:1428] Setting key ff18ed2419c17de6830dc6cc9fa69eb5
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:59.378235 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:59.358099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35385 (local address 127.0.97.62:43997)
0620 02:28:59.358547 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:59.358552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:59.359384 (+ 832us) server_negotiation.cc:407] Connection header received
0620 02:28:59.361323 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:59.361329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:59.361425 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:59.361554 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:59.363126 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.363967 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.366089 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.366731 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.366876 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:59.370780 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:59.370789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:59.370794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:59.370845 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:59.374363 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:59.375542 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:59.375547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:59.375549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:59.375709 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:59.376272 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:59.376276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:59.376279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:59.376571 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:59.376705 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:59.377757 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0620 02:28:59.377960 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":239,"threads_started":1}
I20260620 02:28:59.379889 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:59.358522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43997 (local address 127.0.97.2:35385)
0620 02:28:59.359220 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:59.359260 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:59.360947 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:59.361808 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:59.361818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:59.362351 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:59.362950 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:59.362967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.364138 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.364143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:59.365890 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:59.365905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.366961 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.366967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:59.367181 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:59.367919 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:59.367952 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:59.370485 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:59.374562 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:59.374576 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:59.374599 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:59.375349 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:59.375837 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:59.375844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:59.375851 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:59.376129 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:59.376707 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:59.376718 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:59.377268 (+ 550us) client_negotiation.cc:769] Sending connection context
0620 02:28:59.377646 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 02:28:59.378273 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":295,"threads_started":1}
I20260620 02:28:59.381603 6360 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43997
I20260620 02:28:59.381982 6360 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:59.382781 6360 heartbeater.cc:511] Master 127.0.97.62:43997 requested a full tablet report, sending...
I20260620 02:28:59.384593 6037 ts_manager.cc:194] Registered new tserver with Master: 9a4308ecc8014b37bbef68a5aee04df5 (127.0.97.2:38237)
I20260620 02:28:59.386003 6037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35385
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:28:59.421185 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43997
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:28:59.756394 6366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:28:59.756713 6366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:28:59.756896 6366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:28:59.764437 6366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:28:59.764575 6366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:28:59.764773 6366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:28:59.775435 6366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:28:59.776822 6366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:28:59.778234 6366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:28:59.792001 6374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:59.791950 6371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:28:59.791952 6372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:28:59.792920 6366 server_base.cc:1034] running on GCE node
I20260620 02:28:59.793671 6366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:28:59.794690 6366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:28:59.795950 6366 hybrid_clock.cc:648] HybridClock initialized: now 1781922539795889 us; error 54 us; skew 500 ppm
I20260620 02:28:59.796293 6366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.803970 6366 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:28:59.808419 6366 webserver.cc:460] Webserver started at http://127.0.97.3:46147/ using document root <none> and password file <none>
I20260620 02:28:59.809329 6366 fs_manager.cc:359] Metadata directory not provided
I20260620 02:28:59.809489 6366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:28:59.809813 6366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:28:59.812844 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "e652bcf032a740b1b89671b8f7777599"
format_stamp: "Formatted at 2026-06-20 02:28:59 on dist-test-slave-r706"
server_key: "5df3a92fcf9182004b9f5d993db21e8e"
server_key_iv: "31556a9f6e857fa8de27cbee8a87cfeb"
server_key_version: "encryptionkey@0"
I20260620 02:28:59.813727 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e652bcf032a740b1b89671b8f7777599"
format_stamp: "Formatted at 2026-06-20 02:28:59 on dist-test-slave-r706"
server_key: "5df3a92fcf9182004b9f5d993db21e8e"
server_key_iv: "31556a9f6e857fa8de27cbee8a87cfeb"
server_key_version: "encryptionkey@0"
I20260620 02:28:59.819198 6366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:28:59.823181 6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:59.824641 6366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:28:59.824857 6366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "e652bcf032a740b1b89671b8f7777599"
format_stamp: "Formatted at 2026-06-20 02:28:59 on dist-test-slave-r706"
server_key: "5df3a92fcf9182004b9f5d993db21e8e"
server_key_iv: "31556a9f6e857fa8de27cbee8a87cfeb"
server_key_version: "encryptionkey@0"
I20260620 02:28:59.825148 6366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:28:59.838706 6226 heartbeater.cc:503] Master 127.0.97.62:43997 was elected leader, sending a full tablet report...
I20260620 02:28:59.845256 6366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:28:59.849764 6366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:28:59.850039 6366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:28:59.851001 6366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:28:59.852404 6366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:28:59.852480 6366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:59.852569 6366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:28:59.852613 6366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:28:59.888170 6366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46801
I20260620 02:28:59.888273 6493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46801 every 8 connection(s)
I20260620 02:28:59.889621 6366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:28:59.894594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6366
I20260620 02:28:59.894766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:28:59.895243 388 external_mini_cluster.cc:1428] Setting key 77d98305e5bba82a61b577b3179834a4
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:28:59.916146 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:59.892379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49483 (local address 127.0.97.62:43997)
0620 02:28:59.892827 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:59.892831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:59.893911 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 02:28:59.896418 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:59.896424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:59.896523 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:59.896680 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:59.898980 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.899736 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.902022 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.902684 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.902818 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:59.907265 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:59.907272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:59.907277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:59.907325 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:59.911450 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:59.912423 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:59.912435 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:59.912437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:59.912772 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:59.913748 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:59.913754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:59.913757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:59.914246 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:59.914451 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:59.915408 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 02:28:59.915857 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":212,"threads_started":1}
I20260620 02:28:59.916867 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:59.892924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43997 (local address 127.0.97.3:49483)
0620 02:28:59.893719 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 02:28:59.893758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:28:59.896069 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:28:59.897098 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:28:59.897108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:28:59.897939 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:28:59.898806 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:59.898829 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.899920 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.899925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:59.901852 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:28:59.901867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.902983 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.902990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:28:59.903340 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:59.904097 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:28:59.904127 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:28:59.906972 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:28:59.911695 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:59.911706 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:59.911730 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:59.912255 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:59.912959 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:28:59.912963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:28:59.912965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:28:59.913360 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:28:59.914465 (+ 1105us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:28:59.914472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:28:59.914924 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 02:28:59.915208 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 02:28:59.915683 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":288,"threads_started":1}
I20260620 02:28:59.918481 6494 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43997
I20260620 02:28:59.918790 6494 heartbeater.cc:464] Registering TS with master...
I20260620 02:28:59.919548 6494 heartbeater.cc:511] Master 127.0.97.62:43997 requested a full tablet report, sending...
I20260620 02:28:59.921124 6037 ts_manager.cc:194] Registered new tserver with Master: e652bcf032a740b1b89671b8f7777599 (127.0.97.3:46801)
I20260620 02:28:59.922073 6037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49483
I20260620 02:28:59.924432 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/test-admin.keytab.
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.955794 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 02:28:59.961347 6502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:59.958586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.97.1:33937)
0620 02:28:59.959257 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:59.959265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:59.959317 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:28:59.959382 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:59.959388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:59.959608 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:59.959744 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:59.960809 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40198: BlockingRecv error: recv got EOF from 127.0.0.1:40198 (error 108)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":357,"threads_started":1}
W20260620 02:28:59.961432 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.963150 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.964726 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:28:59.986459 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:28:59.966997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.97.1:33937)
0620 02:28:59.967244 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:59.967249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:59.967311 (+ 62us) server_negotiation.cc:407] Connection header received
0620 02:28:59.967527 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:59.967532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:59.967617 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:59.967711 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:59.968931 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.970022 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.971307 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:28:59.971909 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:28:59.972012 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:28:59.978038 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:28:59.978065 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:28:59.978092 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:28:59.978156 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:28:59.982242 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:59.983726 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:59.983739 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:59.983741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:59.983965 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:28:59.984492 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:28:59.984497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:28:59.984499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:28:59.984955 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:28:59.985351 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 02:28:59.985700 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 02:28:59.986020 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260620 02:28:59.993494 6502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:28:59.991391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.97.1:33937)
0620 02:28:59.991735 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 02:28:59.991741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:28:59.991945 (+ 204us) server_negotiation.cc:407] Connection header received
0620 02:28:59.992162 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:28:59.992167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:28:59.992232 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:28:59.992344 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:28:59.993240 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40202: BlockingRecv error: recv got EOF from 127.0.0.1:40202 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
W20260620 02:28:59.994103 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc)
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.996510 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:28:59 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:28:59.998485 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:29:00.017529 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:00.000674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.97.1:33937)
0620 02:29:00.001109 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:00.001118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:00.001617 (+ 499us) server_negotiation.cc:407] Connection header received
0620 02:29:00.001749 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:00.001753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:00.001831 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:00.001940 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:00.003029 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.003761 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.005168 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.005781 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.005907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:00.010540 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:00.010547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:00.010550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:00.010597 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:00.013649 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.014503 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.014507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.014510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.014662 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.015402 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.015407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.015409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.015724 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:00.015937 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:00.016884 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 02:29:00.017185 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260620 02:29:00.024418 6502 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:00.021247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.97.1:33937)
0620 02:29:00.021720 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:00.021743 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:00.022080 (+ 337us) server_negotiation.cc:407] Connection header received
0620 02:29:00.022436 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:00.022453 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:00.022560 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:00.022672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:00.024132 (+ 1460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40216: BlockingRecv error: recv got EOF from 127.0.0.1:40216 (error 108)
Metrics: {"server-negotiator.queue_time_us":282}
W20260620 02:29:00.024528 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922485472743-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:00.026479 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:00.028057 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:29:00.046192 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:00.029861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.97.1:33937)
0620 02:29:00.030103 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:00.030109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:00.030225 (+ 116us) server_negotiation.cc:407] Connection header received
0620 02:29:00.030380 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:00.030385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:00.030460 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:00.030552 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:00.031662 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.032427 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.033828 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.034264 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.034380 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:00.039264 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:00.039272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:00.039277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:00.039356 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:00.042898 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.043745 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.043752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.043756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.043941 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.044429 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.044434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.044437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.044782 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:00.045116 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:00.045546 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 02:29:00.045851 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:29:00.048208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6098
I20260620 02:29:00.065295 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:00.049623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.0.97.1:33937)
0620 02:29:00.049841 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:00.049849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:00.050289 (+ 440us) server_negotiation.cc:407] Connection header received
0620 02:29:00.050465 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:00.050470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:00.050533 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:00.050620 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:00.051636 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.052338 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.053762 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.054197 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.054311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:00.058260 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:00.058267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:00.058271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:00.058320 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:00.061369 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.062620 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.062625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.062628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.062813 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.063475 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.063479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.063481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.063861 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:00.064099 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:00.064603 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 02:29:00.064890 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:29:00.065970 6221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:00.159734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6098
I20260620 02:29:00.177847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6232
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:29:00.198379 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:00.178982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.97.2:38237)
0620 02:29:00.179663 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:00.179671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:00.179702 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:29:00.179778 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:00.179786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:00.180025 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:00.180177 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:00.181566 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.182675 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.184478 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.185239 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.185372 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:00.190052 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:00.190070 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:00.190088 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:00.190228 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:00.193671 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.194683 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.194692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.194695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.194863 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.195798 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.195802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.195805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.196648 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:00.196943 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:00.197663 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 02:29:00.198010 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":390,"threads_started":1}
I20260620 02:29:00.199916 6355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:00.288463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6232
I20260620 02:29:00.306607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6366
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:29:00.325261 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:00.307771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.97.3:46801)
0620 02:29:00.308600 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:00.308607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:00.308638 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:29:00.308701 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:00.308708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:00.308941 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:00.309166 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:00.310321 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.311416 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.312852 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:00.313767 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:00.313889 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:00.318771 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:00.318787 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:00.318804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:00.318850 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:00.322043 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.322914 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.322923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.322926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.323067 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:00.323476 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:00.323481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:00.323491 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:00.323868 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:00.324208 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:00.324547 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 02:29:00.324834 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":389,"threads_started":1}
I20260620 02:29:00.326735 6489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:00.415184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6366
I20260620 02:29:00.433640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6005
I20260620 02:29:00.434976 6067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:00.540971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6005
2026-06-20T02:29:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4181 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[6522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[6522](info): set up 2 sockets
Jun 20 02:29:00 dist-test-slave-r706 krb5kdc[6522](info): commencing operation
krb5kdc: starting...
W20260620 02:29:01.722024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:29:01 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:29:01.969303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45531
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:02.367544 6535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:02.367909 6535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:02.368019 6535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:02.377072 6535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:02.377236 6535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:02.377287 6535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:02.377341 6535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:02.377388 6535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:02.388093 6535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:02.389490 6535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:02.390818 6535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:02.404232 6541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:02.404184 6540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:02.404517 6543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:02.405368 6535 server_base.cc:1034] running on GCE node
I20260620 02:29:02.406318 6535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:02.408208 6535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:02.409519 6535 hybrid_clock.cc:648] HybridClock initialized: now 1781922542409436 us; error 74 us; skew 500 ppm
I20260620 02:29:02.410003 6535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:02 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922542, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:02.413864 6535 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:29:02.415767 6535 webserver.cc:460] Webserver started at http://127.0.97.62:40857/ using document root <none> and password file <none>
I20260620 02:29:02.416554 6535 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:02.416693 6535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:02.416977 6535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:02.420468 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "9ffa152e2c2a4d54adeaa220f087eee8"
format_stamp: "Formatted at 2026-06-20 02:29:02 on dist-test-slave-r706"
server_key: "4c3716d2d7cfa2943c53cb371f54db5c"
server_key_iv: "0a71f91b2f89ff469e20883a465eeb24"
server_key_version: "encryptionkey@0"
I20260620 02:29:02.421439 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ffa152e2c2a4d54adeaa220f087eee8"
format_stamp: "Formatted at 2026-06-20 02:29:02 on dist-test-slave-r706"
server_key: "4c3716d2d7cfa2943c53cb371f54db5c"
server_key_iv: "0a71f91b2f89ff469e20883a465eeb24"
server_key_version: "encryptionkey@0"
I20260620 02:29:02.427666 6535 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260620 02:29:02.431951 6550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:02.434399 6535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:29:02.434640 6535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "9ffa152e2c2a4d54adeaa220f087eee8"
format_stamp: "Formatted at 2026-06-20 02:29:02 on dist-test-slave-r706"
server_key: "4c3716d2d7cfa2943c53cb371f54db5c"
server_key_iv: "0a71f91b2f89ff469e20883a465eeb24"
server_key_version: "encryptionkey@0"
I20260620 02:29:02.434873 6535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:02.452241 6535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:02.458096 6535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:02.458490 6535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:02.479617 6535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45531
I20260620 02:29:02.479638 6601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45531 every 8 connection(s)
I20260620 02:29:02.481397 6535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:02.485553 6602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:02.486755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6535
I20260620 02:29:02.486961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:29:02.487677 388 external_mini_cluster.cc:1428] Setting key 661d3cf8fde588be1679e11d357ef176
I20260620 02:29:02.501717 6602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Bootstrap starting.
Jun 20 02:29:02 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:02.505281 6602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:02.506855 6602 log.cc:824] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:02.510991 6602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: No bootstrap required, opened a new log
I20260620 02:29:02.515388 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:02.490053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.0.97.62:45531)
0620 02:29:02.491024 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:02.491042 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:02.491104 (+ 62us) server_negotiation.cc:407] Connection header received
0620 02:29:02.493082 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:02.493118 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:02.493753 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:02.494280 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:02.495827 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:02.497946 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:02.499488 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:02.500674 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:02.500886 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:02.505414 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:02.505428 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:02.505450 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:02.505513 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:02.510264 (+ 4751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:02.511142 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:02.511151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:02.511161 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:02.511388 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:02.511804 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:02.511810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:02.511813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:02.512582 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:02.512883 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:02.513367 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 02:29:02.513888 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":422,"threads_started":1}
I20260620 02:29:02.520918 6602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } }
I20260620 02:29:02.521505 6602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:02.521633 6602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffa152e2c2a4d54adeaa220f087eee8, State: Initialized, Role: FOLLOWER
I20260620 02:29:02.522356 6602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } }
I20260620 02:29:02.522653 6602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:02.522773 6602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:02.522904 6602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:02.525400 6602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } }
I20260620 02:29:02.526057 6602 leader_election.cc:302] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ffa152e2c2a4d54adeaa220f087eee8; no voters:
I20260620 02:29:02.526808 6602 leader_election.cc:288] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:29:02.526995 6607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:02.527923 6607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 LEADER]: Becoming Leader. State: Replica: 9ffa152e2c2a4d54adeaa220f087eee8, State: Running, Role: LEADER
I20260620 02:29:02.528503 6607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } }
I20260620 02:29:02.529188 6602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:02.531442 6609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ffa152e2c2a4d54adeaa220f087eee8. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } }
I20260620 02:29:02.532163 6608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } }
I20260620 02:29:02.532406 6609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:02.532474 6608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:02.533743 6613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:02.539165 6613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:02.550661 6613 catalog_manager.cc:1347] Generated new cluster ID: a5a78761f7ee46fbae3a1f0e39d7de34
I20260620 02:29:02.550809 6613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:02.604843 6613 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:29:02.606364 6613 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:02.615612 6613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Generated new TSK 0
I20260620 02:29:02.616806 6613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:29:02.685537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45531
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:03.044883 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:03.045310 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:03.045490 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:03.054069 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:03.054205 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:03.054404 6628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:03.064716 6628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:03.066208 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:03.067626 6628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:03.082413 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:03.082460 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:03.082351 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:03.083389 6628 server_base.cc:1034] running on GCE node
I20260620 02:29:03.084064 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:03.085158 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:03.086426 6628 hybrid_clock.cc:648] HybridClock initialized: now 1781922543086363 us; error 62 us; skew 500 ppm
I20260620 02:29:03.086839 6628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:03 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922543, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:03.090338 6628 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:29:03.091988 6628 webserver.cc:460] Webserver started at http://127.0.97.1:42025/ using document root <none> and password file <none>
I20260620 02:29:03.092684 6628 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:03.092792 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:03.093133 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:03.096128 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "78c77e3ecfbc452996a662ee3602130e"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "980f1e1632acfd38cb785a7cc47600f6"
server_key_iv: "790dd9f4da4820392314afdbc1501836"
server_key_version: "encryptionkey@0"
I20260620 02:29:03.096889 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "78c77e3ecfbc452996a662ee3602130e"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "980f1e1632acfd38cb785a7cc47600f6"
server_key_iv: "790dd9f4da4820392314afdbc1501836"
server_key_version: "encryptionkey@0"
I20260620 02:29:03.102312 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:29:03.106323 6643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:03.107959 6628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:29:03.108167 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "78c77e3ecfbc452996a662ee3602130e"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "980f1e1632acfd38cb785a7cc47600f6"
server_key_iv: "790dd9f4da4820392314afdbc1501836"
server_key_version: "encryptionkey@0"
I20260620 02:29:03.108428 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:03.148576 6628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:03.153308 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:03.153589 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:03.154578 6628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:03.155912 6628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:03.155988 6628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:03.156072 6628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:03.156116 6628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:03.193226 6628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34705
I20260620 02:29:03.193279 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34705 every 8 connection(s)
I20260620 02:29:03.194638 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:03.201285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6628
I20260620 02:29:03.201542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:29:03.202200 388 external_mini_cluster.cc:1428] Setting key b225343c1886d712e1527056ee5c2adc
Jun 20 02:29:03 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922543, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:03.218721 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:03.197526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42293 (local address 127.0.97.62:45531)
0620 02:29:03.198041 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:03.198046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:03.198891 (+ 845us) server_negotiation.cc:407] Connection header received
0620 02:29:03.200784 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:03.200790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:03.200888 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:03.201036 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:03.203009 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.204230 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.206015 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.206456 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.206614 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:03.210819 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:03.210825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:03.210828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:03.210865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:03.214924 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:03.215843 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:03.215852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:03.215858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:03.216021 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:03.216540 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:03.216548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:03.216554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:03.216976 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:03.217168 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:03.217940 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 02:29:03.218210 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":300,"threads_started":1}
I20260620 02:29:03.219388 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:03.197977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:42293)
0620 02:29:03.198716 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:03.198754 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:03.200475 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:03.201301 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:03.201309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:03.201957 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:03.202790 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:03.202810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.204426 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.204437 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:03.205834 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:03.205848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.206725 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.206732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:03.206981 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:03.207725 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:03.207749 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:03.210499 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:03.215125 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:03.215133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:03.215148 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:03.215673 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:03.216181 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:03.216185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:03.216188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:03.216392 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:03.217141 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:03.217147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:03.217493 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 02:29:03.217759 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 02:29:03.218177 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":286,"threads_started":1}
I20260620 02:29:03.221347 6756 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45531
I20260620 02:29:03.221747 6756 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:03.222750 6756 heartbeater.cc:511] Master 127.0.97.62:45531 requested a full tablet report, sending...
I20260620 02:29:03.225348 6567 ts_manager.cc:194] Registered new tserver with Master: 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:29:03.226919 6567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:29:03.262096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45531
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:03.609911 6762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:03.610296 6762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:03.610473 6762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:03.618595 6762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:03.618798 6762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:03.618981 6762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:03.629618 6762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:03.631163 6762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:03.632547 6762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:03.647198 6767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:03.647272 6768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:03.647233 6770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:03.648054 6762 server_base.cc:1034] running on GCE node
I20260620 02:29:03.648695 6762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:03.649813 6762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:03.651067 6762 hybrid_clock.cc:648] HybridClock initialized: now 1781922543650976 us; error 74 us; skew 500 ppm
I20260620 02:29:03.651472 6762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:03 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922543, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:03.655583 6762 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:29:03.657469 6762 webserver.cc:460] Webserver started at http://127.0.97.2:38217/ using document root <none> and password file <none>
I20260620 02:29:03.658232 6762 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:03.658380 6762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:03.658739 6762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:03.662058 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "cb94b6e7171547909fb9523a79f5a0f3"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "b8fbcaee5fc9daa83f7431834d24c3e9"
server_key_iv: "f5ca8866519dadeeffcb3b96e4d42f0a"
server_key_version: "encryptionkey@0"
I20260620 02:29:03.662968 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cb94b6e7171547909fb9523a79f5a0f3"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "b8fbcaee5fc9daa83f7431834d24c3e9"
server_key_iv: "f5ca8866519dadeeffcb3b96e4d42f0a"
server_key_version: "encryptionkey@0"
I20260620 02:29:03.669329 6762 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:29:03.673573 6777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:03.675298 6762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:29:03.675559 6762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "cb94b6e7171547909fb9523a79f5a0f3"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "b8fbcaee5fc9daa83f7431834d24c3e9"
server_key_iv: "f5ca8866519dadeeffcb3b96e4d42f0a"
server_key_version: "encryptionkey@0"
I20260620 02:29:03.675879 6762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:03.698911 6762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:03.704360 6762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:03.704797 6762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:03.706033 6762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:03.708029 6762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:03.708168 6762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:03.708349 6762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:03.708436 6762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:03.755422 6762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33577
I20260620 02:29:03.755542 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33577 every 8 connection(s)
I20260620 02:29:03.756897 6762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:03.758049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6762
I20260620 02:29:03.758236 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:29:03.758822 388 external_mini_cluster.cc:1428] Setting key 92d1e0c475e3f082155e1ba9670ee9c3
Jun 20 02:29:03 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922543, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:29:03.787904 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:03.760152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50593 (local address 127.0.97.62:45531)
0620 02:29:03.760707 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:03.760711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:03.762716 (+ 2005us) server_negotiation.cc:407] Connection header received
0620 02:29:03.766260 (+ 3544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:03.766265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:03.766357 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:03.766465 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:03.769333 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.770324 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.773171 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.774070 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.774274 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:03.778922 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:03.778934 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:03.778943 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:03.779005 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:03.783452 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:03.784522 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:03.784531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:03.784538 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:03.784759 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:03.785456 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:03.785464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:03.785467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:03.785932 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:03.786154 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:03.787073 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 02:29:03.787375 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":330,"threads_started":1}
I20260620 02:29:03.789446 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:03.761302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:50593)
0620 02:29:03.762492 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:03.762550 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:03.765753 (+ 3203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:03.766812 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:03.766829 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:03.767934 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:03.768963 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:03.769035 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.770878 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.770888 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:03.772854 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:03.772873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:03.774425 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:03.774431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:03.774704 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:03.775579 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:03.775611 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:03.778519 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:03.783718 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:03.783728 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:03.783745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:03.784319 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:03.784936 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:03.784940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:03.784943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:03.785292 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:03.786145 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:03.786153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:03.786610 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 02:29:03.786871 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 02:29:03.787337 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":482,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260620 02:29:03.791785 6890 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45531
I20260620 02:29:03.792181 6890 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:03.793119 6890 heartbeater.cc:511] Master 127.0.97.62:45531 requested a full tablet report, sending...
I20260620 02:29:03.795043 6567 ts_manager.cc:194] Registered new tserver with Master: cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577)
I20260620 02:29:03.795993 6567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:29:03.827354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45531
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:04.208431 6896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:04.208847 6896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:04.209126 6896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:04.217741 6896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:04.217936 6896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:04.218120 6896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:04.228557 6896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:04.230165 6896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:04.231179 6756 heartbeater.cc:503] Master 127.0.97.62:45531 was elected leader, sending a full tablet report...
I20260620 02:29:04.232048 6896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:04.253549 6901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:04.253588 6902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:04.253450 6904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:04.253893 6896 server_base.cc:1034] running on GCE node
I20260620 02:29:04.254804 6896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:04.256150 6896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:04.257561 6896 hybrid_clock.cc:648] HybridClock initialized: now 1781922544257389 us; error 103 us; skew 500 ppm
I20260620 02:29:04.258190 6896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:04 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:04.264024 6896 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:29:04.266225 6896 webserver.cc:460] Webserver started at http://127.0.97.3:34783/ using document root <none> and password file <none>
I20260620 02:29:04.267127 6896 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:04.267263 6896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:04.267654 6896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:04.271238 6896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "6ca2b15f70634207a7b438742e69b302"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "d73033d5f30f5d2d8b0c53ffa5659f98"
server_key_iv: "2efac182762afcd6f543287e98c8b4ce"
server_key_version: "encryptionkey@0"
I20260620 02:29:04.272264 6896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6ca2b15f70634207a7b438742e69b302"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "d73033d5f30f5d2d8b0c53ffa5659f98"
server_key_iv: "2efac182762afcd6f543287e98c8b4ce"
server_key_version: "encryptionkey@0"
I20260620 02:29:04.279263 6896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:29:04.284031 6911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:04.285988 6896 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:29:04.286281 6896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "6ca2b15f70634207a7b438742e69b302"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "d73033d5f30f5d2d8b0c53ffa5659f98"
server_key_iv: "2efac182762afcd6f543287e98c8b4ce"
server_key_version: "encryptionkey@0"
I20260620 02:29:04.286550 6896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:04.304275 6896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:04.310691 6896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:04.311102 6896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:04.312526 6896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:04.314296 6896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:04.314428 6896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:04.314604 6896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:04.314689 6896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:04.359530 6896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44331
I20260620 02:29:04.359575 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44331 every 8 connection(s)
I20260620 02:29:04.361084 6896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:04.366816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 6896
I20260620 02:29:04.367048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:29:04.367566 388 external_mini_cluster.cc:1428] Setting key fd1a19ffd9257707a12679d58f4fb5b2
Jun 20 02:29:04 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:04.385337 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:04.364263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44433 (local address 127.0.97.62:45531)
0620 02:29:04.364782 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:04.364787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:04.365813 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 02:29:04.368173 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:04.368179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:04.368303 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:04.368487 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:04.370321 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.371128 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.373278 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.373781 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.373966 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:04.378206 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:04.378212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:04.378215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:04.378256 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:04.381498 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:04.382632 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:04.382638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:04.382646 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:04.382788 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:04.383296 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:04.383301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:04.383309 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:04.383621 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:04.383799 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:04.384611 (+ 812us) server_negotiation.cc:299] Negotiation successful
0620 02:29:04.384887 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":274,"threads_started":1}
I20260620 02:29:04.387035 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:04.364738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:44433)
0620 02:29:04.365614 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:04.365661 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:04.367788 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:04.368759 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:04.368769 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:04.369473 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:04.370124 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:04.370144 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.371323 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.371333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:04.372974 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:04.372992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.374088 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.374096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:04.374376 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:04.375175 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:04.375202 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:04.377872 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:04.381717 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:04.381728 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:04.381753 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:04.382479 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:04.382921 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:04.382926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:04.382930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:04.383154 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:04.383768 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:04.383776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:04.384186 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 02:29:04.384485 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:29:04.385147 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":423,"threads_started":1}
I20260620 02:29:04.388814 7024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45531
I20260620 02:29:04.389374 7024 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:04.390332 7024 heartbeater.cc:511] Master 127.0.97.62:45531 requested a full tablet report, sending...
I20260620 02:29:04.392382 6567 ts_manager.cc:194] Registered new tserver with Master: 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331)
I20260620 02:29:04.393404 6567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44433
I20260620 02:29:04.396957 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 02:29:04.453668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46423
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45531,127.0.97.61:46423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:29:04.799139 6890 heartbeater.cc:503] Master 127.0.97.62:45531 was elected leader, sending a full tablet report...
W20260620 02:29:04.835801 7030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:04.836136 7030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:04.836230 7030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:04.844303 7030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:04.844432 7030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:04.844480 7030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:04.844532 7030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:04.844578 7030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:04.855085 7030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45531,127.0.97.61:46423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
W20260620 02:29:04.855465 7030 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:45531,127.0.97.61:46423'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:29:04.856530 7030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:04.857997 7030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:04.871279 7036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:04.871343 7035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:04.871362 7038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:04.872359 7030 server_base.cc:1034] running on GCE node
I20260620 02:29:04.873431 7030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:04.875178 7030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:04.876502 7030 hybrid_clock.cc:648] HybridClock initialized: now 1781922544876441 us; error 44 us; skew 500 ppm
I20260620 02:29:04.876959 7030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:04 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:04.884737 7030 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:29:04.886662 7030 webserver.cc:460] Webserver started at http://127.0.97.61:39811/ using document root <none> and password file <none>
I20260620 02:29:04.887404 7030 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:04.887509 7030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:04.887914 7030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:04.891443 7030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/instance:
uuid: "76105f54940c45e1ad5bb2478c7e4766"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "5a689445c897ef3e3e1f1175ae5666fd"
server_key_iv: "cca17d2410ecaead02dbded04941fa44"
server_key_version: "encryptionkey@0"
I20260620 02:29:04.892453 7030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal/instance:
uuid: "76105f54940c45e1ad5bb2478c7e4766"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "5a689445c897ef3e3e1f1175ae5666fd"
server_key_iv: "cca17d2410ecaead02dbded04941fa44"
server_key_version: "encryptionkey@0"
I20260620 02:29:04.898437 7030 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:29:04.902652 7045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:04.904295 7030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260620 02:29:04.904536 7030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
uuid: "76105f54940c45e1ad5bb2478c7e4766"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "5a689445c897ef3e3e1f1175ae5666fd"
server_key_iv: "cca17d2410ecaead02dbded04941fa44"
server_key_version: "encryptionkey@0"
I20260620 02:29:04.904942 7030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:04.938236 7030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:04.944502 7030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:04.944881 7030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:04.969900 7030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46423
I20260620 02:29:04.969920 7096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46423 every 8 connection(s)
I20260620 02:29:04.971419 7030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:29:04.975392 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:04.980227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 7030
I20260620 02:29:04.983440 7097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:29:04 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:29:04 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:29:05.006312 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:04.986644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.97.62:45531)
0620 02:29:04.987209 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:04.987217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:04.987905 (+ 688us) server_negotiation.cc:407] Connection header received
0620 02:29:04.987980 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:04.987985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:04.988067 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:04.988180 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:04.989994 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.991175 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.993655 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.994293 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.994459 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:04.999008 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:04.999014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:04.999017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:04.999064 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.002511 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.003694 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.003698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.003701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.003820 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.004291 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.004296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.004299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.004644 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.004827 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.005658 (+ 831us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.005927 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
I20260620 02:29:05.007464 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:04.982819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.0.97.61:46423)
0620 02:29:04.983708 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:04.983720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:04.983762 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:29:04.986638 (+ 2876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:04.986676 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:04.987336 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:04.987746 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:04.989739 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.991510 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.993486 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.994339 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.994527 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:04.999767 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:04.999782 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:04.999803 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:04.999852 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.003472 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.004214 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.004221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.004224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.004350 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.004764 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.004769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.004777 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.005239 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.005576 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.005905 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.006306 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":662,"spinlock_wait_cycles":12672,"thread_start_us":447,"threads_started":1}
I20260620 02:29:05.007504 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:04.986589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44116)
0620 02:29:04.987536 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:04.987578 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:04.987782 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:04.988289 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:04.988296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:04.989225 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:04.989780 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:04.989798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.991339 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.991344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:04.993488 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:04.993502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:04.994639 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:04.994647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:04.994835 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:04.995682 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:04.995717 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:04.998747 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.002700 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.002710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.002721 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.003528 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.003959 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.003965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.003968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.004140 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.004792 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.004801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.005269 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.005558 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.006144 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"spinlock_wait_cycles":15104,"thread_start_us":554,"threads_started":1}
I20260620 02:29:05.009660 7097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922544, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:29:05.023231 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.010528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48006)
0620 02:29:05.010760 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.010781 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.010900 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.011217 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.011221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.011606 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.011732 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.011741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.012775 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.012782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.013822 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.013835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.014570 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.014575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.014824 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.015547 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.015565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.017863 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.021040 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.021048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.021053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.021487 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.021828 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.021831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.021833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.021972 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.022491 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.022494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.022618 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.022727 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.022974 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":1}
I20260620 02:29:05.023449 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.010652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.0.97.61:46423)
0620 02:29:05.010810 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.010818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.010836 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:05.011031 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.011036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.011099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.011203 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.011867 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.012643 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.013976 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.014413 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.014561 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.018075 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.018079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.018083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.018113 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.020804 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.021605 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.021609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.021611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.021725 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.022072 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.022075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.022077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.022360 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.022484 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.022826 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.023113 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":60}
I20260620 02:29:05.028426 7097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Bootstrap starting.
I20260620 02:29:05.032202 7097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:05.033427 7097 log.cc:824] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:05.036509 7097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: No bootstrap required, opened a new log
I20260620 02:29:05.044348 7097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } }
I20260620 02:29:05.044764 7097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:05.044878 7097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76105f54940c45e1ad5bb2478c7e4766, State: Initialized, Role: FOLLOWER
I20260620 02:29:05.045696 7097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } }
I20260620 02:29:05.046777 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } } }
I20260620 02:29:05.047174 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:05.047951 7097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:29:05.058249 7115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:29:05.058486 7115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:29:05.060559 7030 master_runner.cc:428] Detected that this master 76105f54940c45e1ad5bb2478c7e4766 is joining an existing cluster
I20260620 02:29:05.060684 7030 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:46423
I20260620 02:29:05.094089 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.070577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.97.62:45531)
0620 02:29:05.070830 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.070838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.071818 (+ 980us) server_negotiation.cc:407] Connection header received
0620 02:29:05.072166 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.072172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.072290 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.072380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.074618 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.076098 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.078778 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.079485 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.079638 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.083384 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.083392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.083397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.083453 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.089208 (+ 5755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.090639 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.090647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.090654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.091031 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.091593 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.091602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.091609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.092283 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.092472 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.093219 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.093563 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:29:05.094166 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.070597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44132)
0620 02:29:05.071596 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.071614 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.071977 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.072631 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.072637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.073672 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.074376 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.074394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.076356 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.076378 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.078573 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.078594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.079904 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.079911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.080111 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.081212 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.081251 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.083046 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.089446 (+ 6400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.089451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.089456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.090155 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.091167 (+ 1012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.091172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.091176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.091419 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.092455 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.092465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.092768 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.093030 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.093445 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":3,"thread_start_us":1805,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:29:05.098335 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.071444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.97.61:46423)
0620 02:29:05.071588 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.071594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.076544 (+ 4950us) server_negotiation.cc:407] Connection header received
0620 02:29:05.076935 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.076942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.077052 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.077130 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.078901 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.080141 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.082940 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.084088 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.084221 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.087913 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.087922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.087930 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.087974 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.092913 (+ 4939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.095611 (+ 2698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.095619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.095622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.095874 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.096421 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.096429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.096436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.096939 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.097143 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.097575 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.097887 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":57}
I20260620 02:29:05.099473 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.072060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48014)
0620 02:29:05.076275 (+ 4215us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.076300 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.076555 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.077362 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.077367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.078249 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.078725 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.078738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.080442 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.080450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.082754 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.082775 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.084645 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.084654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.084920 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.085748 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.085776 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.087686 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.094607 (+ 6921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.094617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.094621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.095454 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.096066 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.096072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.096082 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.096304 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.097146 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.097153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.097320 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.097479 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.098564 (+ 1085us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4044,"mutex_wait_us":669,"thread_start_us":4559,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922542, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:29:05.125692 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.102519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.0.97.61:46423)
0620 02:29:05.102715 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.102723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.103810 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 02:29:05.104228 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.104235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.104357 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.104478 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.106214 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.107363 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.110455 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.111356 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.111557 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.117352 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.117361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.117367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.117413 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.121445 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.122812 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.122818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.122822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.123037 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.123673 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.123678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.123681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.124086 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.124259 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.124991 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.125303 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:29:05.125880 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.102575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48022)
0620 02:29:05.103505 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.103584 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.103952 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.104562 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.104573 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.105614 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.105981 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.105998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.107579 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.107594 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.110226 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.110250 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.111861 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.111868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.112151 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.113164 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.113201 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.116957 (+ 3756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.121673 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.121696 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.121711 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.122579 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.123218 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.123228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.123234 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.123493 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.124271 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.124283 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.124576 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.124838 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.125330 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":368,"threads_started":1}
I20260620 02:29:05.128453 6567 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:46423: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 46423
}
attrs {
promote: true
}
}
dest_uuid: "9ffa152e2c2a4d54adeaa220f087eee8"
cas_config_opid_index: -1
I20260620 02:29:05.129896 6567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } }
I20260620 02:29:05.134089 7128 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:05.135658 7072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 02:29:05.138048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44151
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:29:05.139127 7128 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:46423
I20260620 02:29:05.139703 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:05.139969 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:05.140079 7030 master.cc:537] Master@127.0.97.61:46423 shutting down...
I20260620 02:29:05.140336 7072 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Refusing update from remote peer 9ffa152e2c2a4d54adeaa220f087eee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:29:05.145437 7128 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:29:05.150173 6554 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 02:29:05.154307 6554 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 -> Peer 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61:46423): Couldn't send request to peer 76105f54940c45e1ad5bb2478c7e4766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:29:05.155999 7030 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:05.156456 7030 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:05.156663 7030 tablet_replica.cc:330] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: stopping tablet replica
I20260620 02:29:05.173820 7030 master.cc:552] Master@127.0.97.61:46423 shutdown complete.
I20260620 02:29:05.173978 7030 master_runner.cc:305] Clearing existing system tablet
I20260620 02:29:05.174396 7030 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:05.174516 7030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:05.176895 7030 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:29:05.178651 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:05.179031 7030 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:05.179349 7030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:29:05.179446 7030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
uuid: "76105f54940c45e1ad5bb2478c7e4766"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "5a689445c897ef3e3e1f1175ae5666fd"
server_key_iv: "cca17d2410ecaead02dbded04941fa44"
server_key_version: "encryptionkey@0"
I20260620 02:29:05.183573 7030 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:05.189652 7030 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:29:05.189879 7030 log.cc:1196] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 02:29:05.190399 7030 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Deleting consensus metadata
I20260620 02:29:05.190692 7030 master_runner.cc:315] Copying system tablet from 127.0.97.62:45531
I20260620 02:29:05.199462 7030 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45531
I20260620 02:29:05.217310 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.200352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.0.97.62:45531)
0620 02:29:05.200595 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.200605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.204824 (+ 4219us) server_negotiation.cc:407] Connection header received
0620 02:29:05.205103 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.205112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.205207 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.205331 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.206241 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.206978 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.208252 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.208752 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.208980 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.211308 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.211315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.211318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.211357 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.214389 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.215123 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.215130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.215134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.215268 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.215687 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.215694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.215697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.216082 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.216248 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.216669 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.216898 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:29:05.217432 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.200305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44140)
0620 02:29:05.204668 (+ 4363us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.204692 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.204924 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.205357 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.205362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.205817 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.206077 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.206088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.207142 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.207148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.208079 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.208091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.209068 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.209077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.209290 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.210192 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.210213 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.211017 (+ 804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.214559 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.214565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.214569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.214985 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.215397 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.215404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.215407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.215563 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.216208 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.216213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.216394 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.216554 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.216941 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4208,"thread_start_us":4088,"threads_started":1}
I20260620 02:29:05.218621 6587 tablet_copy_service.cc:140] P 9ffa152e2c2a4d54adeaa220f087eee8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 76105f54940c45e1ad5bb2478c7e4766 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44140)
I20260620 02:29:05.218916 6587 tablet_copy_service.cc:161] P 9ffa152e2c2a4d54adeaa220f087eee8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 76105f54940c45e1ad5bb2478c7e4766 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44140: session id = 76105f54940c45e1ad5bb2478c7e4766-00000000000000000000000000000000
I20260620 02:29:05.222510 6587 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:29:05.224725 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:29:05.230041 7030 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: tablet copy: Starting download of 0 data blocks...
I20260620 02:29:05.230402 7030 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: tablet copy: Starting download of 1 WAL segments...
I20260620 02:29:05.233258 7030 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:29:05.237531 6587 tablet_copy_service.cc:342] P 9ffa152e2c2a4d54adeaa220f087eee8: Request end of tablet copy session 76105f54940c45e1ad5bb2478c7e4766-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44140
I20260620 02:29:05.237792 6587 tablet_copy_service.cc:433] P 9ffa152e2c2a4d54adeaa220f087eee8: ending tablet copy session 76105f54940c45e1ad5bb2478c7e4766-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 76105f54940c45e1ad5bb2478c7e4766
I20260620 02:29:05.249740 7030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:05.252592 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:05.258808 7141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:05.267524 7030 server_base.cc:1034] running on GCE node
W20260620 02:29:05.267812 7143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:05.268342 7030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:05.269078 7030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:05.270401 7030 hybrid_clock.cc:648] HybridClock initialized: now 1781922545270329 us; error 38 us; skew 500 ppm
I20260620 02:29:05.270697 7030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:05.272801 7030 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:29:05.278095 7030 webserver.cc:460] Webserver started at http://127.0.97.61:37093/ using document root <none> and password file <none>
I20260620 02:29:05.278486 7030 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:05.278600 7030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:05.281782 7030 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 02:29:05.283669 7150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:05.284126 7030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260620 02:29:05.284293 7030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
uuid: "76105f54940c45e1ad5bb2478c7e4766"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "5a689445c897ef3e3e1f1175ae5666fd"
server_key_iv: "cca17d2410ecaead02dbded04941fa44"
server_key_version: "encryptionkey@0"
I20260620 02:29:05.284476 7030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:05.300772 7030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:05.304490 7030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:05.343909 7030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46423
I20260620 02:29:05.343933 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46423 every 8 connection(s)
I20260620 02:29:05.344933 7030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:29:05.352573 7202 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:29:05.358368 7202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Bootstrap starting.
I20260620 02:29:05.378700 7202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:29:05.379290 7202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: Bootstrap complete.
I20260620 02:29:05.380409 7202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } }
I20260620 02:29:05.380761 7202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 76105f54940c45e1ad5bb2478c7e4766, State: Initialized, Role: LEARNER
I20260620 02:29:05.381062 7202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } }
I20260620 02:29:05.382397 7202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:05.382297 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:05.382795 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:29:05.390694 7216 catalog_manager.cc:1259] Loaded cluster ID: a5a78761f7ee46fbae3a1f0e39d7de34
I20260620 02:29:05.390823 7216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: loading cluster ID for follower catalog manager: success
I20260620 02:29:05.394354 7216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: acquiring CA information for follower catalog manager: success
I20260620 02:29:05.396693 7216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:29:05.397270 7024 heartbeater.cc:503] Master 127.0.97.62:45531 was elected leader, sending a full tablet report...
I20260620 02:29:05.647231 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.633675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48030)
0620 02:29:05.634853 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.634896 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.635090 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.636939 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.636945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.636977 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:05.637473 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.637488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.640947 (+ 3459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.640955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.643725 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.643747 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.646233 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.646241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.646433 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.646501 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.646572 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.646716 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":875,"threads_started":1}
I20260620 02:29:05.650157 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.635175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.0.97.61:46423)
0620 02:29:05.636308 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.636314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.636386 (+ 72us) server_negotiation.cc:407] Connection header received
0620 02:29:05.636486 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.636497 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.636639 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.637661 (+ 1022us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:05.638051 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.640700 (+ 2649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.644041 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.646003 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.648800 (+ 2797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.649243 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.649311 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.649477 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":632,"threads_started":1}
I20260620 02:29:05.652369 7177 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 02:29:05.653698 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ffa152e2c2a4d54adeaa220f087eee8. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:05.654109 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: LEARNER
W20260620 02:29:05.750542 7131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:05.750980 7131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:05.751195 7131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:05.762997 7131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:05.763248 7131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:05.763399 7131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:05.763561 7131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:05.763685 7131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:05.776671 7131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:05.778383 7131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:05.779963 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:05.801522 7223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:05.801947 7224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:05.807590 7226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:05.809731 7131 server_base.cc:1034] running on GCE node
I20260620 02:29:05.810918 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:05.813246 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:05.814764 7131 hybrid_clock.cc:648] HybridClock initialized: now 1781922545814530 us; error 199 us; skew 500 ppm
I20260620 02:29:05.815292 7131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:05.820379 7131 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:29:05.823244 7131 webserver.cc:460] Webserver started at http://127.0.97.60:38027/ using document root <none> and password file <none>
I20260620 02:29:05.824205 7131 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:05.824426 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:05.824853 7131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:05.829128 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/instance:
uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
format_stamp: "Formatted at 2026-06-20 02:29:05 on dist-test-slave-r706"
server_key: "28a64d17cf54e7af1aea39bd75cbaa1c"
server_key_iv: "823ab92f5a017243530d07e82b5118ae"
server_key_version: "encryptionkey@0"
I20260620 02:29:05.830304 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal/instance:
uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
format_stamp: "Formatted at 2026-06-20 02:29:05 on dist-test-slave-r706"
server_key: "28a64d17cf54e7af1aea39bd75cbaa1c"
server_key_iv: "823ab92f5a017243530d07e82b5118ae"
server_key_version: "encryptionkey@0"
I20260620 02:29:05.837805 7131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:29:05.843488 7233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:05.845373 7131 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:29:05.845631 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
format_stamp: "Formatted at 2026-06-20 02:29:05 on dist-test-slave-r706"
server_key: "28a64d17cf54e7af1aea39bd75cbaa1c"
server_key_iv: "823ab92f5a017243530d07e82b5118ae"
server_key_version: "encryptionkey@0"
I20260620 02:29:05.845906 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:05.866555 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:05.872612 7131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:05.873138 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:05.915099 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44151
I20260620 02:29:05.915253 7284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44151 every 8 connection(s)
I20260620 02:29:05.916956 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:29:05.922147 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:05.926502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 7131
I20260620 02:29:05.931984 7285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:05.962656 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.938331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.0.97.62:45531)
0620 02:29:05.939109 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.939114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.940966 (+ 1852us) server_negotiation.cc:407] Connection header received
0620 02:29:05.941331 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.941337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.941518 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.941738 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.943903 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.944992 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.947142 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.948147 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.948371 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.953803 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.953809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.953812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.953855 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.957879 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.959651 (+ 1772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.959657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.959665 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.959850 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.960445 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.960449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.960451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.960938 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.961154 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.961839 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.962154 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":526,"threads_started":1}
I20260620 02:29:05.963157 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.929540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.97.60:44151)
0620 02:29:05.931039 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.931052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.931110 (+ 58us) server_negotiation.cc:407] Connection header received
0620 02:29:05.933594 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.933623 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.934197 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.934773 (+ 576us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.937545 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.941412 (+ 3867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.943351 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.944487 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.944627 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.951928 (+ 7301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.951944 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.951963 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.952029 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.956350 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.957801 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.957810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.957818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.958088 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.958725 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.958729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.958732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.960023 (+ 1291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.960301 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.961049 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.961609 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":1229,"thread_start_us":878,"threads_started":1}
I20260620 02:29:05.963255 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.938564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44156)
0620 02:29:05.940672 (+ 2108us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.940733 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.941168 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.941873 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.941879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.943065 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.943717 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.943731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.945215 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.945219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.946905 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.946921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.948601 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.948610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.948817 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.949766 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.949807 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.953506 (+ 3699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.958087 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.958100 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.958108 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.959479 (+ 1371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.960001 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.960006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.960010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.960232 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.961144 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.961152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.961424 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.961670 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.961928 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1627,"mutex_wait_us":4,"thread_start_us":1121,"threads_started":1}
I20260620 02:29:05.965174 7285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:29:05.982837 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.966049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48038)
0620 02:29:05.966294 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.966315 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.966463 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.967266 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.967270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.967766 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.968009 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.968019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.969426 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.969431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.970607 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.970623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.971639 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.971644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.971918 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.972664 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.972683 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.975411 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.979676 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.979680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.979684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.980318 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.980838 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.980842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.980844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.981176 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.981775 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:05.981778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:05.981944 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 02:29:05.982181 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 02:29:05.982489 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 02:29:05.982898 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.966141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.97.61:46423)
0620 02:29:05.966362 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.966367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.966385 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:05.966629 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.966642 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.966782 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.966892 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.968254 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.969231 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.970797 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.971393 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.971521 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.975716 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.975723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.975726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.975767 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.979445 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.980487 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.980492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.980494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.980666 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.981290 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.981294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.981296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.981666 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:05.981786 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:05.982286 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 02:29:05.982557 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:29:05.984243 7285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:29:05 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922545, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:29:06.000941 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.985580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.0.1:41764)
0620 02:29:05.985742 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:05.985763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:05.985871 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:05.986219 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:05.986222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:05.986725 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:05.986922 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.986931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.988046 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.988050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.989303 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:05.989319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.990410 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.990415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:05.990640 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.991505 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:05.991524 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:05.994296 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:05.998207 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.998212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.998214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.998706 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:05.999168 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:05.999172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:05.999174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:05.999414 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.000035 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:06.000038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:06.000191 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.000302 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.000522 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:29:06.000960 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:05.985303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.97.60:44151)
0620 02:29:05.985552 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:05.985557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:05.985824 (+ 267us) server_negotiation.cc:407] Connection header received
0620 02:29:05.985987 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:05.985990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:05.986048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:05.986185 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:05.987049 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.987903 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.989485 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:05.990211 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:05.990363 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:05.994560 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:05.994566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:05.994569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:05.994601 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:05.998027 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.998842 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.998846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.998848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.999024 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:05.999565 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:05.999569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:05.999571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:05.999923 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.000042 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.000452 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.000633 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 02:29:06.007472 7285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Bootstrap starting.
I20260620 02:29:06.012147 7285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:06.014225 7285 log.cc:824] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:06.018592 7285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: No bootstrap required, opened a new log
I20260620 02:29:06.028687 7285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } }
I20260620 02:29:06.029493 7285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:06.029767 7285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e5b2f9b5b74d9d84f0c0059dd09ece, State: Initialized, Role: FOLLOWER
I20260620 02:29:06.030614 7285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } }
I20260620 02:29:06.032024 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } } }
I20260620 02:29:06.032589 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:06.033428 7285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:29:06.050046 7303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:29:06.050284 7303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:29:06.055101 7131 master_runner.cc:428] Detected that this master 54e5b2f9b5b74d9d84f0c0059dd09ece is joining an existing cluster
I20260620 02:29:06.055341 7131 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44151
I20260620 02:29:06.071405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6628
I20260620 02:29:06.091032 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:06.091948 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.100067 7127 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: attempting to promote NON_VOTER 76105f54940c45e1ad5bb2478c7e4766 to VOTER
I20260620 02:29:06.101145 7127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } }
I20260620 02:29:06.103829 7177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 LEARNER]: Refusing update from remote peer 9ffa152e2c2a4d54adeaa220f087eee8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:29:06.105496 7314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
Jun 20 02:29:06 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 02:29:06.122512 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.061997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.0.97.62:45531)
0620 02:29:06.062236 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.062242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.078565 (+ 16323us) server_negotiation.cc:407] Connection header received
0620 02:29:06.079030 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.079040 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.079165 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.079266 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:06.082575 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.084397 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.088800 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.089920 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.090062 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.104224 (+ 14162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:06.104232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:06.104236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:06.104281 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:06.112705 (+ 8424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.117577 (+ 4872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.117584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.117598 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.118202 (+ 604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.118992 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.118997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.119001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.119529 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.121452 (+ 1923us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.121666 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.122023 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:29:06.125481 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.062586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44166)
0620 02:29:06.078239 (+ 15653us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.078283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:06.078784 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:06.079635 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:06.079644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:06.081088 (+ 1444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:06.082190 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.082211 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.085802 (+ 3591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.085808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.088521 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.088539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.090324 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.090332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.090696 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.102423 (+ 11727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:06.102449 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:06.103914 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:06.113508 (+ 9594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.113515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.113520 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.114748 (+ 1228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.118381 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.118389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.118393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.118783 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.119721 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:06.119731 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:06.120084 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.120269 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.120732 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15336,"mutex_wait_us":1,"spinlock_wait_cycles":46592,"thread_start_us":4069,"threads_started":1}
W20260620 02:29:06.126032 7239 tablet.cc:2369] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:06.128409 7313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } }
I20260620 02:29:06.131217 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.062276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.97.60:44151)
0620 02:29:06.062549 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.062555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.080032 (+ 17477us) server_negotiation.cc:407] Connection header received
0620 02:29:06.080945 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.080954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.081058 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.081148 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:06.086928 (+ 5780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.088850 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.099725 (+ 10875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.100742 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.100924 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.109721 (+ 8797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:06.109728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:06.109731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:06.109768 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:06.118402 (+ 8634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.119529 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.119534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.119538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.119759 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.120260 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.120264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.120267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.120857 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.121092 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.130155 (+ 9063us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.130479 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:29:06.132027 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.063245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.0.1:41778)
0620 02:29:06.079781 (+ 16536us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.079831 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:06.080574 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:06.081467 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:06.081475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:06.083045 (+ 1570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:06.086515 (+ 3470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.086538 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.089206 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.089215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.099525 (+ 10310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.099547 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.101802 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.101818 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.102145 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.107903 (+ 5758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:06.107938 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:06.109504 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:06.118579 (+ 9075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.118596 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.118604 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.119374 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.119931 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.119936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.119939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.120127 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.121200 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:06.121208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:06.129732 (+ 8524us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.130625 (+ 893us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.131310 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15302,"spinlock_wait_cycles":1280,"thread_start_us":34984,"threads_started":1}
I20260620 02:29:06.129769 7177 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } }
I20260620 02:29:06.135820 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } }
I20260620 02:29:06.136324 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.136794 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } }
I20260620 02:29:06.137105 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.144542 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.063389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48046)
0620 02:29:06.100164 (+ 36775us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.100187 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:06.100485 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:06.101421 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:06.101425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:06.105589 (+ 4164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:06.106793 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.106811 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.111552 (+ 4741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.111562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.115240 (+ 3678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.115267 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.117134 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.117147 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.117426 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.118414 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:06.118561 (+ 147us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:06.120252 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:06.130408 (+ 10156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.130414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.130418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.134631 (+ 4213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.139854 (+ 5223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.139860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.139864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.140161 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.141100 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:06.141105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:06.143081 (+ 1976us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.143241 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.143739 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36570,"spinlock_wait_cycles":99712,"thread_start_us":19575,"threads_started":1}
I20260620 02:29:06.146173 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.062074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.0.97.61:46423)
0620 02:29:06.062276 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.062282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.101063 (+ 38781us) server_negotiation.cc:407] Connection header received
0620 02:29:06.101149 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.101155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.101283 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.101396 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:06.107127 (+ 5731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.108577 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.115458 (+ 6881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.116490 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.116663 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.121692 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:06.121700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:06.121704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:06.121752 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:06.129424 (+ 7672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.138064 (+ 8640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.138072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.138076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.139621 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.140333 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.140339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.140342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.140783 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.140983 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.145194 (+ 4211us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.145548 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 02:29:06 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922542, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:29:06.150539 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.073177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.0.97.1:34705)
0620 02:29:06.086514 (+ 13337us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.086524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.086570 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:29:06.086705 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.086716 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.087071 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.094694 (+ 7623us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:06.095152 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.097812 (+ 2660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.102280 (+ 4468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.103823 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.104014 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.125588 (+ 21574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:06.127354 (+ 1766us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:06.127378 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:06.127471 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:06.144671 (+ 17200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.146045 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.146057 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.146061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.146333 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.146950 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.146956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.146968 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.147556 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.147984 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.149375 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.149896 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13038,"thread_start_us":12390,"threads_started":1}
I20260620 02:29:06.150838 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } }
I20260620 02:29:06.151186 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:06.152444 6751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:06.161228 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.136928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.0.1:41788)
0620 02:29:06.137375 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.137397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:06.137582 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:06.138437 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:06.138445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:06.139182 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:06.139462 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.139472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.141333 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.141339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.142658 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.142671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.144380 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.144394 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.144673 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.146038 (+ 1365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:06.146075 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:06.149843 (+ 3768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:06.157412 (+ 7569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.157419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.157424 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.158043 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.158636 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.158652 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.158656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.158913 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.159748 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:06.159755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:06.160103 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.160338 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.160725 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260620 02:29:06.161293 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.137830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.0.97.60:44151)
0620 02:29:06.138003 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.138010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.138026 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:29:06.138158 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.138165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.138298 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.138409 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:06.139622 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.141162 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.142866 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.143905 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.144053 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.150241 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:06.150248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:06.150252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:06.150293 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:06.157222 (+ 6929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.158231 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.158240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.158246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.158453 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.159077 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.159083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.159088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.159576 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.159752 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.160487 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.160807 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260620 02:29:06.163365 6567 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44151: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44151
}
attrs {
promote: true
}
}
dest_uuid: "9ffa152e2c2a4d54adeaa220f087eee8"
cas_config_opid_index: 6
I20260620 02:29:06.164419 6567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } }
I20260620 02:29:06.167889 7177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Refusing update from remote peer 9ffa152e2c2a4d54adeaa220f087eee8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:29:06.168584 7260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:06.168776 7313 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:29:06.172307 7127 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.174057 7260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Refusing update from remote peer 9ffa152e2c2a4d54adeaa220f087eee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:29:06.175165 7314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:06.178238 7176 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.183815 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.184288 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:06.184748 7127 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44151
I20260620 02:29:06.185679 7131 master.cc:537] Master@127.0.97.60:44151 shutting down...
I20260620 02:29:06.187849 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.188086 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.187429 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.188352 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.203389 7259 raft_consensus.cc:858] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } }
W20260620 02:29:06.203667 7259 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } }
I20260620 02:29:06.226020 7259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:06.233948 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ffa152e2c2a4d54adeaa220f087eee8. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:06.234653 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 02:29:06.233920 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:06.235085 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 02:29:06.233120 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: true } } }
I20260620 02:29:06.235828 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 02:29:06.237133 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.237448 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.238708 7131 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 02:29:06.239282 7131 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 02:29:06.239512 7131 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: stopping tablet replica
W20260620 02:29:06.241641 6552 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 -> Peer 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60:44151): Couldn't send request to peer 54e5b2f9b5b74d9d84f0c0059dd09ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:29:06.247524 7131 master.cc:552] Master@127.0.97.60:44151 shutdown complete.
I20260620 02:29:06.247659 7131 master_runner.cc:305] Clearing existing system tablet
I20260620 02:29:06.248144 7131 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:06.248308 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:06.251133 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 02:29:06.252764 7326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:06.253196 7131 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:06.253507 7131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 02:29:06.253618 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
format_stamp: "Formatted at 2026-06-20 02:29:05 on dist-test-slave-r706"
server_key: "28a64d17cf54e7af1aea39bd75cbaa1c"
server_key_iv: "823ab92f5a017243530d07e82b5118ae"
server_key_version: "encryptionkey@0"
I20260620 02:29:06.257447 7131 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:06.263429 7131 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:29:06.263713 7131 log.cc:1196] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 02:29:06.264317 7131 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Deleting consensus metadata
I20260620 02:29:06.264632 7131 master_runner.cc:315] Copying system tablet from 127.0.97.62:45531
I20260620 02:29:06.270695 7131 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45531
I20260620 02:29:06.285283 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6628
I20260620 02:29:06.292068 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.271640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.0.97.62:45531)
0620 02:29:06.272000 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.272009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.272888 (+ 879us) server_negotiation.cc:407] Connection header received
0620 02:29:06.273215 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.273220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.273320 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.273445 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:06.274514 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.275307 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.276907 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.277613 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.277788 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.280257 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:06.280270 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:06.280274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:06.280311 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:06.286963 (+ 6652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.288073 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.288082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.288088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.288331 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:06.288918 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:06.288927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:06.288934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:06.289479 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:06.289643 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.291263 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.291600 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260620 02:29:06.292843 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.271545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44180)
0620 02:29:06.272754 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.272774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:06.272984 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:06.273477 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:06.273481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:06.274059 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:06.274353 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.274363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.275523 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.275528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.276730 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.276743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.277796 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.277802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.278086 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.278996 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:06.279016 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:06.280002 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:06.287153 (+ 7151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.287162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.287170 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.287897 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.288487 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:06.288496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:06.288501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:06.288772 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:06.289712 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:06.289720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:06.290078 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.290243 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.290802 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":790,"threads_started":1}
I20260620 02:29:06.294073 6587 tablet_copy_service.cc:140] P 9ffa152e2c2a4d54adeaa220f087eee8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 54e5b2f9b5b74d9d84f0c0059dd09ece ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44180)
I20260620 02:29:06.294286 6587 tablet_copy_service.cc:161] P 9ffa152e2c2a4d54adeaa220f087eee8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 54e5b2f9b5b74d9d84f0c0059dd09ece at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44180: session id = 54e5b2f9b5b74d9d84f0c0059dd09ece-00000000000000000000000000000000
I20260620 02:29:06.298806 6587 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:29:06.301067 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:29:06.309576 7131 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: tablet copy: Starting download of 0 data blocks...
I20260620 02:29:06.309952 7131 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: tablet copy: Starting download of 1 WAL segments...
I20260620 02:29:06.310891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34705
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45531
--webserver_port=42025
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:29:06.312402 7131 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:29:06.316046 6587 tablet_copy_service.cc:342] P 9ffa152e2c2a4d54adeaa220f087eee8: Request end of tablet copy session 54e5b2f9b5b74d9d84f0c0059dd09ece-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44180
I20260620 02:29:06.316216 6587 tablet_copy_service.cc:433] P 9ffa152e2c2a4d54adeaa220f087eee8: ending tablet copy session 54e5b2f9b5b74d9d84f0c0059dd09ece-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 54e5b2f9b5b74d9d84f0c0059dd09ece
I20260620 02:29:06.329397 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:06.332381 7334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:06.338652 7335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:06.346477 7131 server_base.cc:1034] running on GCE node
W20260620 02:29:06.349821 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:06.350333 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:06.364853 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:06.372442 7131 hybrid_clock.cc:648] HybridClock initialized: now 1781922546372386 us; error 64 us; skew 500 ppm
I20260620 02:29:06.372807 7131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:06 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:06.375018 7131 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:29:06.376650 7131 webserver.cc:460] Webserver started at http://127.0.97.60:35963/ using document root <none> and password file <none>
I20260620 02:29:06.376978 7131 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:06.377140 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:06.379688 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 02:29:06.381539 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:06.381973 7131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 02:29:06.382118 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
format_stamp: "Formatted at 2026-06-20 02:29:05 on dist-test-slave-r706"
server_key: "28a64d17cf54e7af1aea39bd75cbaa1c"
server_key_iv: "823ab92f5a017243530d07e82b5118ae"
server_key_version: "encryptionkey@0"
I20260620 02:29:06.382342 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:06.399806 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:06.403841 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:06.438181 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44151
I20260620 02:29:06.438247 7395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44151 every 8 connection(s)
I20260620 02:29:06.439545 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:29:06.447930 7396 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:29:06.452180 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Bootstrap starting.
I20260620 02:29:06.464219 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:29:06.464759 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Bootstrap complete.
I20260620 02:29:06.465744 7396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } }
I20260620 02:29:06.466142 7396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54e5b2f9b5b74d9d84f0c0059dd09ece, State: Initialized, Role: LEARNER
I20260620 02:29:06.466526 7396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } }
I20260620 02:29:06.467864 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.468271 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: LEARNER
I20260620 02:29:06.468509 7396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:06.498893 7410 catalog_manager.cc:1259] Loaded cluster ID: a5a78761f7ee46fbae3a1f0e39d7de34
I20260620 02:29:06.499033 7410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: loading cluster ID for follower catalog manager: success
I20260620 02:29:06.502661 7410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: acquiring CA information for follower catalog manager: success
I20260620 02:29:06.505437 7410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:29:06.712387 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.712738 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.723272 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.712969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.0.1:41794)
0620 02:29:06.714648 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.714684 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:06.714913 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:06.716290 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:06.716296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:06.716337 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:06.716628 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.716640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.718555 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.718561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.720311 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:06.720327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.722033 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.722041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:06.722368 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.722404 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:29:06.722524 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:29:06.722712 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":2534,"threads_started":1}
I20260620 02:29:06.723240 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.713130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.0.97.60:44151)
0620 02:29:06.715843 (+ 2713us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.715848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.715879 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:29:06.715979 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.715984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.716088 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.716259 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:06.716816 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.718388 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.720494 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:06.721789 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:06.721963 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:06.722315 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:06.722551 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 02:29:06.722730 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2515,"thread_start_us":2292,"threads_started":1}
I20260620 02:29:06.724537 7371 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260620 02:29:06.725116 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ffa152e2c2a4d54adeaa220f087eee8. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: true } } }
I20260620 02:29:06.725473 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: LEARNER
I20260620 02:29:06.726068 7314 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: attempting to promote NON_VOTER 54e5b2f9b5b74d9d84f0c0059dd09ece to VOTER
I20260620 02:29:06.727502 7314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:06.730679 7177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Refusing update from remote peer 9ffa152e2c2a4d54adeaa220f087eee8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:29:06.730988 7371 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 LEARNER]: Refusing update from remote peer 9ffa152e2c2a4d54adeaa220f087eee8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:29:06.731546 7314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:29:06.731980 7313 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:29:06.735711 7313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.736690 7177 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.739621 7371 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.741401 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.741879 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.742249 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.742669 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:06.749392 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.749809 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:06.751567 7415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.752295 7415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:06.752636 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:06.753069 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8 [sys.catalog]: This master's current role is: LEADER
W20260620 02:29:06.783911 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:06.784260 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:06.784451 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:06.794159 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:06.794318 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:06.794468 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:06.806531 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42025
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:06.808076 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:06.809875 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:06.830998 7420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:06.831321 7423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:06.831784 7421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:06.834007 7333 server_base.cc:1034] running on GCE node
I20260620 02:29:06.834779 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:06.836037 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:06.837337 7333 hybrid_clock.cc:648] HybridClock initialized: now 1781922546837277 us; error 56 us; skew 500 ppm
I20260620 02:29:06.837744 7333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:06 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:06.841961 7333 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:29:06.844297 7333 webserver.cc:460] Webserver started at http://127.0.97.1:42025/ using document root <none> and password file <none>
I20260620 02:29:06.845351 7333 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:06.845578 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:06.852831 7333 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 02:29:06.856330 7430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:06.857900 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:29:06.858124 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "78c77e3ecfbc452996a662ee3602130e"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "980f1e1632acfd38cb785a7cc47600f6"
server_key_iv: "790dd9f4da4820392314afdbc1501836"
server_key_version: "encryptionkey@0"
I20260620 02:29:06.859405 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:06.896951 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:06.903349 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:06.903815 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:06.905397 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:06.907392 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:06.907600 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:06.907797 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:06.907959 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:06.984139 7333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34705
I20260620 02:29:06.984479 7542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34705 every 8 connection(s)
I20260620 02:29:06.986037 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:06.994310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 7333
I20260620 02:29:06.994828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6762
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:07.026422 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.996241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.0.97.2:33577)
0620 02:29:06.997444 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.997457 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.997498 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:29:06.997614 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:06.997627 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:06.997953 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:06.998205 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.001641 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.004066 (+ 2425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.006451 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.007974 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.008122 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.015741 (+ 7619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.015762 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.015782 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.015846 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.020501 (+ 4655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.021803 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.021810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.021812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.022107 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.022789 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.022793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.022796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.023298 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.023678 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.025338 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.025712 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":642,"threads_started":1}
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:29:07.028565 6885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:07.036240 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.993485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.97.1:43373)
0620 02:29:06.995110 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.995129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:07.004652 (+ 9523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:07.005783 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:07.005795 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:07.007914 (+ 2119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:07.009344 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.009373 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.011585 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.011592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.015309 (+ 3717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.015333 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.016816 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.016825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.017263 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.018322 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:07.018376 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:07.022876 (+ 4500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:07.029913 (+ 7037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.029925 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.029993 (+ 68us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.031395 (+ 1402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.031944 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.031950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.031956 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.032278 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.033250 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:07.033260 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:07.033829 (+ 569us) client_negotiation.cc:769] Sending connection context
0620 02:29:07.034207 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 02:29:07.034769 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490,"mutex_wait_us":197,"spinlock_wait_cycles":896,"thread_start_us":8579,"threads_started":1}
I20260620 02:29:07.037916 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.990088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43373 (local address 127.0.97.61:46423)
0620 02:29:06.991048 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.991056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.995309 (+ 4253us) server_negotiation.cc:407] Connection header received
0620 02:29:07.005152 (+ 9843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.005161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.005280 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.005429 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.009684 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.011361 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.015539 (+ 4178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.016198 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.016294 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.023264 (+ 6970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.023270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.023274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.023310 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.029702 (+ 6392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.031577 (+ 1875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.031583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.031588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.031799 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.032405 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.032412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.032417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.032983 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.036764 (+ 3781us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.036934 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.037360 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":554,"threads_started":1}
I20260620 02:29:07.038424 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.990805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:50797)
0620 02:29:06.992440 (+ 1635us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:06.992501 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:07.004652 (+ 12151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:07.006677 (+ 2025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:07.006690 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:07.007904 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:07.010265 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.010274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.013523 (+ 3249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.013529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.015519 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.015532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.017194 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.017201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.017429 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.020660 (+ 3231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:07.020684 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:07.025908 (+ 5224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:07.032551 (+ 6643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.032561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.032568 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.033513 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.034147 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.034152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.034156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.034414 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.036072 (+ 1658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:07.036087 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:07.036377 (+ 290us) client_negotiation.cc:769] Sending connection context
0620 02:29:07.037420 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0620 02:29:07.037781 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":116,"thread_start_us":3020,"threads_started":1}
I20260620 02:29:07.039763 7545 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46423
I20260620 02:29:07.040174 7545 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:07.040522 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.992035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50797 (local address 127.0.97.62:45531)
0620 02:29:06.992826 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.992836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:06.992865 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:29:07.005821 (+ 12956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.005828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.005996 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.006190 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.011631 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.013321 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.015711 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.016794 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.016962 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.026376 (+ 9414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.026391 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.026405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.026462 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.031192 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.033764 (+ 2572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.033770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.033772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.033971 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.035404 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.035408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.035410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.035872 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.036094 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.039679 (+ 3585us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.040017 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":1770,"threads_started":1}
I20260620 02:29:07.041785 7545 heartbeater.cc:511] Master 127.0.97.61:46423 requested a full tablet report, sending...
I20260620 02:29:07.043008 7544 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45531
I20260620 02:29:07.043205 7544 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:07.043478 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.990016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37625 (local address 127.0.97.60:44151)
0620 02:29:06.991783 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:06.991797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:07.005485 (+ 13688us) server_negotiation.cc:407] Connection header received
0620 02:29:07.005765 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.005771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.005855 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.005958 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.010491 (+ 4533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.011329 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.019332 (+ 8003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.020685 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.020848 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.034705 (+ 13857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.034712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.034715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.034752 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.039081 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.040377 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.040385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.040391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.040726 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.041367 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.041372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.041375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.041889 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.042077 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.042641 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.042987 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564}
I20260620 02:29:07.043982 7544 heartbeater.cc:511] Master 127.0.97.62:45531 requested a full tablet report, sending...
I20260620 02:29:07.044715 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:06.993593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.97.1:37625)
0620 02:29:07.005335 (+ 11742us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:07.005352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:07.005617 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:07.006286 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:07.006291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:07.007904 (+ 1613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:07.010244 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.010257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.011585 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.011592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.019120 (+ 7528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.019135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.021711 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.021720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.022848 (+ 1128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.023790 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:07.023825 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:07.034361 (+ 10536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:07.039359 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.039365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.039370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.040127 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.040920 (+ 793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.040926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.040929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.041187 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.042165 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:07.042170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:07.042321 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 02:29:07.043687 (+ 1366us) client_negotiation.cc:240] Negotiation successful
0620 02:29:07.044057 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11606,"mutex_wait_us":4943,"spinlock_wait_cycles":88192,"thread_start_us":1716,"threads_started":1}
I20260620 02:29:07.045156 7167 ts_manager.cc:194] Registered new tserver with Master: 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705)
I20260620 02:29:07.045854 6567 ts_manager.cc:194] Re-registered known tserver with Master: 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705)
I20260620 02:29:07.046132 7543 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44151
I20260620 02:29:07.046309 7543 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:07.047024 6567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50797
I20260620 02:29:07.047045 7543 heartbeater.cc:511] Master 127.0.97.60:44151 requested a full tablet report, sending...
I20260620 02:29:07.049502 7361 ts_manager.cc:194] Registered new tserver with Master: 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705)
I20260620 02:29:07.127590 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6762
I20260620 02:29:07.145210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33577
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45531
--webserver_port=38217
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:07.491778 7555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:07.492125 7555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:07.492313 7555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:07.499611 7555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:07.499809 7555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:07.500011 7555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:07.510152 7555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38217
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:07.511615 7555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:07.512979 7555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:07.527040 7561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:07.526954 7563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:07.527524 7560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:07.528400 7555 server_base.cc:1034] running on GCE node
I20260620 02:29:07.529088 7555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:07.530105 7555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:07.531371 7555 hybrid_clock.cc:648] HybridClock initialized: now 1781922547531322 us; error 38 us; skew 500 ppm
I20260620 02:29:07.531769 7555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:07.535070 7555 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:29:07.536522 7555 webserver.cc:460] Webserver started at http://127.0.97.2:38217/ using document root <none> and password file <none>
I20260620 02:29:07.537410 7555 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:07.537585 7555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:07.543569 7555 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 02:29:07.546741 7570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:07.548169 7555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:29:07.548362 7555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "cb94b6e7171547909fb9523a79f5a0f3"
format_stamp: "Formatted at 2026-06-20 02:29:03 on dist-test-slave-r706"
server_key: "b8fbcaee5fc9daa83f7431834d24c3e9"
server_key_iv: "f5ca8866519dadeeffcb3b96e4d42f0a"
server_key_version: "encryptionkey@0"
I20260620 02:29:07.549501 7555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:07.571893 7555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:07.576872 7555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:07.577286 7555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:07.578451 7555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:07.579969 7555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:07.580065 7555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:07.580209 7555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:07.580295 7555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:07.616072 7555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33577
I20260620 02:29:07.616176 7682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33577 every 8 connection(s)
I20260620 02:29:07.617689 7555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:07.619006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 7555
I20260620 02:29:07.619300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6896
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:29:07.664517 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.620849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38803 (local address 127.0.97.61:46423)
0620 02:29:07.631289 (+ 10440us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:07.631300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:07.637427 (+ 6127us) server_negotiation.cc:407] Connection header received
0620 02:29:07.637509 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.637515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.637623 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.637835 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.641572 (+ 3737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.642790 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.645981 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.646812 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.646947 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.653581 (+ 6634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.653598 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.653603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.653687 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.658630 (+ 4943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.660127 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.660137 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.660143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.660429 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.661327 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.661335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.661341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.661898 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.662094 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.663553 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.663908 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10248,"thread_start_us":16390,"threads_started":1}
Jun 20 02:29:07 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:29:07.665565 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.621321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51555 (local address 127.0.97.60:44151)
0620 02:29:07.631707 (+ 10386us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:07.631715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:07.637662 (+ 5947us) server_negotiation.cc:407] Connection header received
0620 02:29:07.637803 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.637810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.637897 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.638090 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.641586 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.642800 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.645986 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.646895 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.647009 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.654925 (+ 7916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.654930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.654933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.655022 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.660760 (+ 5738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.662207 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.662213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.662216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.662430 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.663045 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.663050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.663056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.663581 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.663762 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.664639 (+ 877us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.664882 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10224,"thread_start_us":10119,"threads_started":1}
I20260620 02:29:07.665982 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.639196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43753 (local address 127.0.97.62:45531)
0620 02:29:07.640453 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:07.640459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:07.641375 (+ 916us) server_negotiation.cc:407] Connection header received
0620 02:29:07.641447 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.641454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.641550 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.641768 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.643108 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.643826 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.646145 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.646729 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.646887 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.655582 (+ 8695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.655588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.655591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.655628 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.660761 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.662201 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.662207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.662211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.662447 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.663110 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.663127 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.663131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.663589 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.663757 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.665206 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.665481 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":999,"threads_started":1}
I20260620 02:29:07.667469 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.622317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.97.2:38803)
0620 02:29:07.632409 (+ 10092us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:07.632478 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:07.636206 (+ 3728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:07.639738 (+ 3532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:07.639743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:07.640546 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:07.641360 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.641380 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.643018 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.643024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.645764 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.645779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.647450 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.647456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.647723 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.649075 (+ 1352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:07.649101 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:07.653121 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:07.658904 (+ 5783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.658920 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.658947 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.659918 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.660651 (+ 733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.660659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.660666 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.661146 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.662241 (+ 1095us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:07.662252 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:07.662929 (+ 677us) client_negotiation.cc:769] Sending connection context
0620 02:29:07.665267 (+ 2338us) client_negotiation.cc:240] Negotiation successful
0620 02:29:07.666168 (+ 901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9737,"mutex_wait_us":226,"spinlock_wait_cycles":3328,"thread_start_us":9457,"threads_started":1}
I20260620 02:29:07.667833 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.622830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.97.2:51555)
0620 02:29:07.637267 (+ 14437us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:07.637297 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:07.637542 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:07.638658 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:07.638681 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:07.640007 (+ 1326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:07.641348 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.641375 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.643018 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.643024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.645764 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.645779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.647413 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.647425 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.647682 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.648689 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:07.648735 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:07.654661 (+ 5926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:07.660943 (+ 6282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.660949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.660953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.662033 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.662584 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.662589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.662593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.662890 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.663825 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:07.663831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:07.664193 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 02:29:07.664373 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 02:29:07.666486 (+ 2113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14271,"mutex_wait_us":2061,"spinlock_wait_cycles":32128,"thread_start_us":14080,"threads_started":1}
I20260620 02:29:07.668028 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.638644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:43753)
0620 02:29:07.640927 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:07.640951 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:07.641252 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:07.641807 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:07.641811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:07.642500 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:07.642952 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.642969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.643991 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.644000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.646000 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:07.646015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.647044 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.647052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:07.647498 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.648580 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:07.648628 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:07.654510 (+ 5882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:07.660943 (+ 6433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.660949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.660953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.662033 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.662584 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:07.662589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:07.662593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:07.662984 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:07.663825 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:07.663831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:07.664074 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 02:29:07.664293 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 02:29:07.664913 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2122,"mutex_wait_us":62,"spinlock_wait_cycles":8192,"thread_start_us":1957,"threads_started":1}
I20260620 02:29:07.669976 7684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45531
I20260620 02:29:07.670285 7685 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46423
I20260620 02:29:07.670472 7684 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:07.670486 7685 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:07.671922 7683 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44151
I20260620 02:29:07.672233 7683 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:07.672259 7684 heartbeater.cc:511] Master 127.0.97.62:45531 requested a full tablet report, sending...
I20260620 02:29:07.674047 7683 heartbeater.cc:511] Master 127.0.97.60:44151 requested a full tablet report, sending...
I20260620 02:29:07.674067 7685 heartbeater.cc:511] Master 127.0.97.61:46423 requested a full tablet report, sending...
I20260620 02:29:07.674537 6567 ts_manager.cc:194] Re-registered known tserver with Master: cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577)
I20260620 02:29:07.676326 7167 ts_manager.cc:194] Registered new tserver with Master: cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577)
I20260620 02:29:07.676306 7361 ts_manager.cc:194] Registered new tserver with Master: cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577)
I20260620 02:29:07.676610 6567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43753
I20260620 02:29:07.679785 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:07.620975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.0.97.3:44331)
0620 02:29:07.645251 (+ 24276us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:07.645262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:07.645291 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:29:07.645355 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:07.645363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:07.645618 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:07.650394 (+ 4776us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:07.650890 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.652871 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.656423 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:07.657880 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:07.658133 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:07.666483 (+ 8350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:07.666504 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:07.666522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:07.666584 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:07.672836 (+ 6252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.674304 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.674315 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.674319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.674588 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:07.675178 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:07.675184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:07.675187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:07.676107 (+ 920us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:07.676796 (+ 689us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:07.678992 (+ 2196us) server_negotiation.cc:299] Negotiation successful
0620 02:29:07.679261 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24033,"thread_start_us":24865,"threads_started":1}
I20260620 02:29:07.681453 7019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:07.770534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6896
I20260620 02:29:07.788703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44331
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45531
--webserver_port=34783
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:29:08.050114 7544 heartbeater.cc:503] Master 127.0.97.62:45531 was elected leader, sending a full tablet report...
W20260620 02:29:08.141644 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:08.142007 7696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:08.142210 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:08.150044 7696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:08.150166 7696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:08.150276 7696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:08.160673 7696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34783
--tserver_master_addrs=127.0.97.62:45531,127.0.97.61:46423,127.0.97.60:44151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:08.162091 7696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:08.163373 7696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:08.178083 7701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:08.178496 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:08.178483 7702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:08.179854 7696 server_base.cc:1034] running on GCE node
I20260620 02:29:08.180585 7696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:08.181841 7696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:08.183131 7696 hybrid_clock.cc:648] HybridClock initialized: now 1781922548183049 us; error 70 us; skew 500 ppm
I20260620 02:29:08.183471 7696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:08 dist-test-slave-r706 krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:08.187584 7696 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:29:08.189646 7696 webserver.cc:460] Webserver started at http://127.0.97.3:34783/ using document root <none> and password file <none>
I20260620 02:29:08.190483 7696 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:08.190656 7696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:08.197576 7696 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:29:08.200964 7711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:08.202632 7696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:29:08.202852 7696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "6ca2b15f70634207a7b438742e69b302"
format_stamp: "Formatted at 2026-06-20 02:29:04 on dist-test-slave-r706"
server_key: "d73033d5f30f5d2d8b0c53ffa5659f98"
server_key_iv: "2efac182762afcd6f543287e98c8b4ce"
server_key_version: "encryptionkey@0"
I20260620 02:29:08.203938 7696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:08.227324 7696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:08.232331 7696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:08.232657 7696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:08.233811 7696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:08.235427 7696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:08.235582 7696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:08.235745 7696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:08.235853 7696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:08.273928 7696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44331
I20260620 02:29:08.273989 7823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44331 every 8 connection(s)
I20260620 02:29:08.275472 7696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:08.283936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 7696
Jun 20 02:29:08 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:29:08 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:29:08 dist-test-slave-r706 krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:29:08.326972 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.280457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.97.3:42065)
0620 02:29:08.281653 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.281721 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.285678 (+ 3957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.295471 (+ 9793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.295476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.296042 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:08.296414 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.296426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.298386 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.298392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.301273 (+ 2881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.301290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.302810 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.302821 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.303274 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.304320 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:08.304368 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:08.308753 (+ 4385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:08.318661 (+ 9908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:08.318677 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:08.318706 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:08.319719 (+ 1013us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:08.320429 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:08.320437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:08.320447 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:08.320902 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:08.322002 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:08.322016 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:08.323414 (+ 1398us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.323861 (+ 447us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.324731 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":255,"thread_start_us":547,"threads_started":1}
I20260620 02:29:08.328903 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.279725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42065 (local address 127.0.97.60:44151)
0620 02:29:08.295144 (+ 15419us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.295151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.295172 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:29:08.295233 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.295238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.295315 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.295450 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.297141 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.298188 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.301484 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.302148 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.302280 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.310747 (+ 8467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.310759 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.310767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.310822 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.318308 (+ 7486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.319942 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.319952 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.319958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.320241 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.321133 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.321141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.321146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.321694 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.321881 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.327766 (+ 5885us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.328147 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15274,"thread_start_us":15818,"threads_started":1}
I20260620 02:29:08.331065 7824 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44151
I20260620 02:29:08.331471 7824 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:08.332885 7824 heartbeater.cc:511] Master 127.0.97.60:44151 requested a full tablet report, sending...
I20260620 02:29:08.334542 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.280829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.97.3:37467)
0620 02:29:08.290488 (+ 9659us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.290519 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.290715 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.292223 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.292247 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.294670 (+ 2423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:08.296366 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.296385 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.298389 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.298393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.301271 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.301290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.309347 (+ 8057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.309354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.309603 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.313811 (+ 4208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:08.313838 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:08.318850 (+ 5012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:08.327925 (+ 9075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:08.327931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:08.327934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:08.328839 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:08.330854 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:08.330859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:08.330863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:08.331311 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:08.333304 (+ 1993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:08.333309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:08.333466 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.333686 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.334010 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9464,"mutex_wait_us":572,"thread_start_us":9146,"threads_started":1}
I20260620 02:29:08.335508 7361 ts_manager.cc:194] Registered new tserver with Master: 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331)
I20260620 02:29:08.335868 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.281381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37467 (local address 127.0.97.61:46423)
0620 02:29:08.291295 (+ 9914us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.291305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.291330 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:29:08.291413 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.291421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.291534 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.291685 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.296604 (+ 4919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.298210 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.303477 (+ 5267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.304397 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.304561 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.321402 (+ 16841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.321413 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.321422 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.321486 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.327573 (+ 6087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.329077 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.329086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.329092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.329465 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.331478 (+ 2013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.331486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.331492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.331952 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.332102 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.335004 (+ 2902us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.335303 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9723,"thread_start_us":9591,"threads_started":1}
I20260620 02:29:08.338174 7826 heartbeater.cc:347] Connected to a master server at 127.0.97.61:46423
I20260620 02:29:08.338378 7826 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:08.339313 7826 heartbeater.cc:511] Master 127.0.97.61:46423 requested a full tablet report, sending...
I20260620 02:29:08.340966 7167 ts_manager.cc:194] Registered new tserver with Master: 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331)
I20260620 02:29:08.346194 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.281563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:36917)
0620 02:29:08.292437 (+ 10874us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.292463 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.292628 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.293144 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.293151 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.294670 (+ 1519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:08.296062 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.296118 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.298486 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.298495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.308681 (+ 10186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.308711 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.310166 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.310176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.310426 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.311385 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:08.311417 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:08.331372 (+ 19955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:08.341073 (+ 9701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:08.341079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:08.341084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:08.342067 (+ 983us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:08.342649 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:08.342654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:08.342667 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:08.342905 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:08.344761 (+ 1856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:08.344774 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:08.345103 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.345247 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.345584 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10696,"mutex_wait_us":62,"spinlock_wait_cycles":31872,"thread_start_us":10470,"threads_started":1}
I20260620 02:29:08.346933 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.280048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36917 (local address 127.0.97.62:45531)
0620 02:29:08.280605 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.280614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.292569 (+ 11955us) server_negotiation.cc:407] Connection header received
0620 02:29:08.292773 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.292782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.292905 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.293102 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.296610 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.297829 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.308917 (+ 11088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.309948 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.310095 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.331731 (+ 21636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.331743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.331752 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.331816 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.340831 (+ 9015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.342254 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.342267 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.342271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.342508 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.343042 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.343048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.343058 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.343598 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.343910 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.346335 (+ 2425us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.346617 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260620 02:29:08.347998 7825 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45531
I20260620 02:29:08.348213 7825 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:08.349100 7825 heartbeater.cc:511] Master 127.0.97.62:45531 requested a full tablet report, sending...
I20260620 02:29:08.350617 6567 ts_manager.cc:194] Re-registered known tserver with Master: 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331)
I20260620 02:29:08.351980 6567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36917
I20260620 02:29:08.367105 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.302930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.0.97.60:44151)
0620 02:29:08.330783 (+ 27853us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.330792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.338356 (+ 7564us) server_negotiation.cc:407] Connection header received
0620 02:29:08.338783 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.338792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.338928 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.339022 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.344318 (+ 5296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.345995 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.351987 (+ 5992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.352831 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.353087 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.358128 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.358138 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.358146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.358201 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.362882 (+ 4681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.364148 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.364155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.364159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.364424 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.365050 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.365058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.365062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.365502 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.365915 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.366317 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.366582 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27704,"thread_start_us":34281,"threads_started":1}
I20260620 02:29:08.371436 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.302057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.97.61:46423)
0620 02:29:08.341843 (+ 39786us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.341847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.342949 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 02:29:08.347509 (+ 4560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.347515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.347610 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.347818 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.352628 (+ 4810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.354940 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.358228 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.358835 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.359014 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.362471 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.362478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.362482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.362519 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.367313 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.368547 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.368554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.368560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.368843 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.369497 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.369503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.369506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.369939 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.370191 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.370603 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.370889 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30901,"thread_start_us":30709,"threads_started":1}
I20260620 02:29:08.372298 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.301040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.0.97.62:45531)
0620 02:29:08.341542 (+ 40502us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.341548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.341566 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:08.341618 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.341623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.341702 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.342063 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.348220 (+ 6157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.349743 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.355187 (+ 5444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.356179 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.356361 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.363276 (+ 6915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.363285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.363291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.363336 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.368556 (+ 5220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.369623 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.369630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.369633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.369830 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.370293 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.370298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.370300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.370759 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.371032 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.371494 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.371760 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40351,"thread_start_us":40689,"threads_started":1}
I20260620 02:29:08.377585 6567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:08.388480 6567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:08.433408 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.412410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.0.97.3:44331)
0620 02:29:08.413696 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.413710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.420983 (+ 7273us) server_negotiation.cc:407] Connection header received
0620 02:29:08.421404 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.421414 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.421727 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.422063 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:08.424450 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.426420 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.429993 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.432006 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.432135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.432364 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.432771 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.432988 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":1162,"threads_started":1}
I20260620 02:29:08.433679 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.411794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.0.97.1:34705)
0620 02:29:08.418356 (+ 6562us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.418370 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.419506 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 02:29:08.421426 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.421432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.421735 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.422034 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:08.423532 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.425696 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.429629 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.431898 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.432090 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.432378 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.433061 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.433287 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6276,"thread_start_us":6084,"threads_started":1}
I20260620 02:29:08.437286 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.412065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44331 (local address 127.0.0.1:36694)
0620 02:29:08.420746 (+ 8681us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.420766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.421168 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.423325 (+ 2157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.423329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.423356 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:08.424170 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.424192 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.426541 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.426547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.429730 (+ 3183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.429745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.432260 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.432267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.432475 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.432529 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.432654 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.436595 (+ 3941us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8451,"mutex_wait_us":3866,"thread_start_us":6828,"threads_started":1}
I20260620 02:29:08.438244 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.411492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34705 (local address 127.0.0.1:43168)
0620 02:29:08.419303 (+ 7811us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.419326 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.420136 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.422520 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.422525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.422557 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:08.423055 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.423070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.425951 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.425964 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.429447 (+ 3483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.429465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.432585 (+ 3120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.432590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.432812 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.432844 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.432948 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.437638 (+ 4690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7632,"mutex_wait_us":4482,"thread_start_us":8812,"threads_started":1}
I20260620 02:29:08.442601 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 88b77dcdcea8453aa058d542eccb8045 (DEFAULT_TABLE table=test-table [id=75712d63361c4697b060ba953d236471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:08.442621 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 88b77dcdcea8453aa058d542eccb8045 (DEFAULT_TABLE table=test-table [id=75712d63361c4697b060ba953d236471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:08.444183 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.419529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33577 (local address 127.0.0.1:33950)
0620 02:29:08.421473 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.421497 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.421751 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.426138 (+ 4387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.426145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.426198 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:08.427743 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.427757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.429996 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.430002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.439752 (+ 9750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.439773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.442715 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.442723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.443014 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.443109 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.443213 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.443390 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"mutex_wait_us":27,"thread_start_us":1441,"threads_started":1}
I20260620 02:29:08.444490 7478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b77dcdcea8453aa058d542eccb8045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:08.444798 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b77dcdcea8453aa058d542eccb8045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:08.445336 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.419938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.0.97.2:33577)
0620 02:29:08.424915 (+ 4977us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.424924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.424954 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:29:08.425151 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.425159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.425421 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.425832 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:08.428096 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.429880 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.440036 (+ 10156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.442010 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.442182 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.442419 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.444538 (+ 2119us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.444814 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4770,"thread_start_us":4657,"threads_started":1}
I20260620 02:29:08.451642 7618 tablet_service.cc:1467] Processing CreateTablet for tablet 88b77dcdcea8453aa058d542eccb8045 (DEFAULT_TABLE table=test-table [id=75712d63361c4697b060ba953d236471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:08.453745 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b77dcdcea8453aa058d542eccb8045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:08.462204 7860 tablet_bootstrap.cc:492] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Bootstrap starting.
I20260620 02:29:08.467584 7860 tablet_bootstrap.cc:654] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:08.469022 7861 tablet_bootstrap.cc:492] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Bootstrap starting.
I20260620 02:29:08.469769 7860 log.cc:824] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:08.474426 7861 tablet_bootstrap.cc:654] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:08.475378 7863 tablet_bootstrap.cc:492] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Bootstrap starting.
I20260620 02:29:08.476306 7860 tablet_bootstrap.cc:492] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: No bootstrap required, opened a new log
I20260620 02:29:08.476708 7860 ts_tablet_manager.cc:1397] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260620 02:29:08.476647 7861 log.cc:824] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:08.480378 7863 tablet_bootstrap.cc:654] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:08.481549 7861 tablet_bootstrap.cc:492] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: No bootstrap required, opened a new log
I20260620 02:29:08.481961 7861 ts_tablet_manager.cc:1397] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260620 02:29:08.482524 7863 log.cc:824] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:08.493319 7860 raft_consensus.cc:348] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.493999 7860 raft_consensus.cc:374] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:08.494187 7860 raft_consensus.cc:729] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca2b15f70634207a7b438742e69b302, State: Initialized, Role: FOLLOWER
I20260620 02:29:08.495262 7860 consensus_queue.cc:260] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.497921 7861 raft_consensus.cc:348] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.498734 7861 raft_consensus.cc:374] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:08.498911 7861 raft_consensus.cc:729] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c77e3ecfbc452996a662ee3602130e, State: Initialized, Role: FOLLOWER
I20260620 02:29:08.499944 7861 consensus_queue.cc:260] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.503296 7860 ts_tablet_manager.cc:1428] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Time spent starting tablet: real 0.026s user 0.011s sys 0.013s
I20260620 02:29:08.503757 7825 heartbeater.cc:503] Master 127.0.97.62:45531 was elected leader, sending a full tablet report...
I20260620 02:29:08.510792 7863 tablet_bootstrap.cc:492] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: No bootstrap required, opened a new log
I20260620 02:29:08.511018 7861 ts_tablet_manager.cc:1428] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Time spent starting tablet: real 0.029s user 0.020s sys 0.008s
I20260620 02:29:08.511191 7863 ts_tablet_manager.cc:1397] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.013s
I20260620 02:29:08.524101 7863 raft_consensus.cc:348] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.524570 7863 raft_consensus.cc:374] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:08.524689 7863 raft_consensus.cc:729] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb94b6e7171547909fb9523a79f5a0f3, State: Initialized, Role: FOLLOWER
I20260620 02:29:08.525434 7863 consensus_queue.cc:260] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.526875 7684 heartbeater.cc:503] Master 127.0.97.62:45531 was elected leader, sending a full tablet report...
I20260620 02:29:08.527616 7863 ts_tablet_manager.cc:1428] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260620 02:29:08.530530 7827 tablet.cc:2369] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:08.586619 7867 raft_consensus.cc:482] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:08.587028 7867 raft_consensus.cc:504] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.589073 7867 leader_election.cc:288] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331), cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577)
I20260620 02:29:08.598496 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.589394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59349 (local address 127.0.97.3:44331)
0620 02:29:08.589622 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.589627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.590710 (+ 1083us) server_negotiation.cc:407] Connection header received
0620 02:29:08.591085 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.591095 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.591222 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.591296 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:08.592276 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.593598 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.595918 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.597306 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.597495 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.597659 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.598016 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.598155 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:29:08.598798 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.589312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44331 (local address 127.0.97.1:59349)
0620 02:29:08.590502 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.590521 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.590835 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.591562 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.591565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.591587 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:08.592030 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.592038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.593764 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.593768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.595688 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.595700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.597637 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.597645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.597818 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.597842 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.597991 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.598144 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":695,"threads_started":1}
I20260620 02:29:08.599027 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.590746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41309 (local address 127.0.97.2:33577)
0620 02:29:08.591077 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.591090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.591761 (+ 671us) server_negotiation.cc:407] Connection header received
0620 02:29:08.592004 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.592010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.592332 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.592539 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:08.593452 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.594406 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.596611 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.597724 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.597904 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.598054 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.598435 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.598562 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:29:08.599443 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.590585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33577 (local address 127.0.97.1:41309)
0620 02:29:08.591562 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:08.591589 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:08.591851 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:08.592630 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:08.592633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:08.592656 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:08.593278 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.593288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.594591 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.594596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.596417 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:08.596429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.598096 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.598104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:08.598262 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.598286 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:29:08.598409 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:29:08.598536 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":632,"threads_started":1}
I20260620 02:29:08.600267 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b77dcdcea8453aa058d542eccb8045" candidate_uuid: "78c77e3ecfbc452996a662ee3602130e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca2b15f70634207a7b438742e69b302" is_pre_election: true
I20260620 02:29:08.600720 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b77dcdcea8453aa058d542eccb8045" candidate_uuid: "78c77e3ecfbc452996a662ee3602130e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb94b6e7171547909fb9523a79f5a0f3" is_pre_election: true
I20260620 02:29:08.601258 7779 raft_consensus.cc:2454] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c77e3ecfbc452996a662ee3602130e in term 0.
I20260620 02:29:08.601504 7638 raft_consensus.cc:2454] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c77e3ecfbc452996a662ee3602130e in term 0.
I20260620 02:29:08.602283 7431 leader_election.cc:302] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ca2b15f70634207a7b438742e69b302, 78c77e3ecfbc452996a662ee3602130e; no voters:
I20260620 02:29:08.602689 7867 raft_consensus.cc:2781] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:08.602926 7867 raft_consensus.cc:482] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:08.603117 7867 raft_consensus.cc:3037] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:08.606074 7867 raft_consensus.cc:504] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.606828 7867 leader_election.cc:288] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [CANDIDATE]: Term 1 election: Requested vote from peers 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331), cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577)
I20260620 02:29:08.607604 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b77dcdcea8453aa058d542eccb8045" candidate_uuid: "78c77e3ecfbc452996a662ee3602130e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca2b15f70634207a7b438742e69b302"
I20260620 02:29:08.607681 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b77dcdcea8453aa058d542eccb8045" candidate_uuid: "78c77e3ecfbc452996a662ee3602130e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb94b6e7171547909fb9523a79f5a0f3"
I20260620 02:29:08.607928 7779 raft_consensus.cc:3037] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:08.607926 7638 raft_consensus.cc:3037] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:08.612705 7638 raft_consensus.cc:2454] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c77e3ecfbc452996a662ee3602130e in term 1.
I20260620 02:29:08.612705 7779 raft_consensus.cc:2454] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c77e3ecfbc452996a662ee3602130e in term 1.
I20260620 02:29:08.613334 7431 leader_election.cc:302] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c77e3ecfbc452996a662ee3602130e, cb94b6e7171547909fb9523a79f5a0f3; no voters:
I20260620 02:29:08.613785 7867 raft_consensus.cc:2781] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:08.614579 7867 raft_consensus.cc:686] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 1 LEADER]: Becoming Leader. State: Replica: 78c77e3ecfbc452996a662ee3602130e, State: Running, Role: LEADER
I20260620 02:29:08.615272 7867 consensus_queue.cc:237] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } }
I20260620 02:29:08.622140 6567 catalog_manager.cc:5495] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c77e3ecfbc452996a662ee3602130e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "78c77e3ecfbc452996a662ee3602130e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } health_report { overall_health: UNKNOWN } } }
W20260620 02:29:08.640666 7686 tablet.cc:2369] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:08.684926 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.678568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.0.97.1:34705)
0620 02:29:08.678800 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.678804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.678822 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:08.678915 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.678919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.678998 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.679139 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:08.679583 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.680480 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.682131 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.682711 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.682898 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.683572 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:08.684186 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:08.684350 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.684417 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.684534 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:29:08.695606 7638 raft_consensus.cc:1261] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Refusing update from remote peer 78c77e3ecfbc452996a662ee3602130e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:08.695603 7779 raft_consensus.cc:1261] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Refusing update from remote peer 78c77e3ecfbc452996a662ee3602130e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:08.696762 7867 consensus_queue.cc:1035] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:08.697224 7871 consensus_queue.cc:1035] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:08.714085 7875 mvcc.cc:205] Tried to move back new op lower bound from 7298754759444164608 to 7298754759132868608. Current Snapshot: MvccSnapshot[applied={T|T < 7298754759444164608}]
I20260620 02:29:08.719337 7876 mvcc.cc:205] Tried to move back new op lower bound from 7298754759444164608 to 7298754759132868608. Current Snapshot: MvccSnapshot[applied={T|T < 7298754759444164608}]
I20260620 02:29:08.743212 6567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:08.743534 6567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:08.751685 6567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ffa152e2c2a4d54adeaa220f087eee8: Sending DeleteTablet for 3 replicas of tablet 88b77dcdcea8453aa058d542eccb8045
I20260620 02:29:08.753170 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b77dcdcea8453aa058d542eccb8045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43168
I20260620 02:29:08.753667 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b77dcdcea8453aa058d542eccb8045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36694
I20260620 02:29:08.755308 7618 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b77dcdcea8453aa058d542eccb8045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33950
I20260620 02:29:08.765225 7886 tablet_replica.cc:330] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: stopping tablet replica
I20260620 02:29:08.766036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 6535
I20260620 02:29:08.765992 7886 raft_consensus.cc:2229] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:08.767735 6597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:08.771029 7889 tablet_replica.cc:330] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: stopping tablet replica
I20260620 02:29:08.767150 7886 raft_consensus.cc:2258] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:08.771680 7889 raft_consensus.cc:2229] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:08.772703 7889 raft_consensus.cc:2258] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:08.773717 7887 tablet_replica.cc:330] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: stopping tablet replica
I20260620 02:29:08.775213 7889 ts_tablet_manager.cc:1905] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:08.775678 7886 ts_tablet_manager.cc:1905] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:08.775671 7887 raft_consensus.cc:2229] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:08.776463 7887 raft_consensus.cc:2258] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:08.782725 7887 ts_tablet_manager.cc:1905] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:08.785851 7889 ts_tablet_manager.cc:1918] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:08.786197 7889 log.cc:1196] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/88b77dcdcea8453aa058d542eccb8045
I20260620 02:29:08.787132 7889 ts_tablet_manager.cc:1939] T 88b77dcdcea8453aa058d542eccb8045 P cb94b6e7171547909fb9523a79f5a0f3: Deleting consensus metadata
I20260620 02:29:08.791571 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.756203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.0.97.61:46423)
0620 02:29:08.756473 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.756482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.764288 (+ 7806us) server_negotiation.cc:407] Connection header received
0620 02:29:08.764621 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.764630 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.764768 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.764907 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.769836 (+ 4929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.771454 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.774647 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.775544 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.775695 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.780801 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.780810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.780818 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.780872 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.786147 (+ 5275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.787745 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.787751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.787756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.787975 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.788579 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.788585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.788588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.789167 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.789448 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.790287 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.790744 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:29:08.793708 7886 ts_tablet_manager.cc:1918] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:08.793977 7886 log.cc:1196] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/88b77dcdcea8453aa058d542eccb8045
I20260620 02:29:08.794728 7886 ts_tablet_manager.cc:1939] T 88b77dcdcea8453aa058d542eccb8045 P 78c77e3ecfbc452996a662ee3602130e: Deleting consensus metadata
I20260620 02:29:08.800612 7887 ts_tablet_manager.cc:1918] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:08.800873 7887 log.cc:1196] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/88b77dcdcea8453aa058d542eccb8045
I20260620 02:29:08.801661 7887 ts_tablet_manager.cc:1939] T 88b77dcdcea8453aa058d542eccb8045 P 6ca2b15f70634207a7b438742e69b302: Deleting consensus metadata
I20260620 02:29:08.818516 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.764337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.97.60:44151)
0620 02:29:08.764553 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.764559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.794933 (+ 30374us) server_negotiation.cc:407] Connection header received
0620 02:29:08.795134 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.795140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.795243 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.795356 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.798453 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.799966 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.802667 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.803502 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.803620 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.808953 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.808963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.808970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.809051 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.813993 (+ 4942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.815313 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.815321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.815326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.815550 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.816074 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.816080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.816084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.816593 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.816898 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.817613 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.817863 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:29:08.930723 6551 catalog_manager.cc:4841] TS cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577): tablet 88b77dcdcea8453aa058d542eccb8045 (table test-table [id=75712d63361c4697b060ba953d236471]) successfully deleted
I20260620 02:29:08.930764 6554 catalog_manager.cc:4841] TS 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705): tablet 88b77dcdcea8453aa058d542eccb8045 (table test-table [id=75712d63361c4697b060ba953d236471]) successfully deleted
W20260620 02:29:08.931056 6551 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:33577, user_credentials={real_user=slave}} blocked reactor thread for 141395us
I20260620 02:29:08.931614 6551 catalog_manager.cc:4841] TS 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331): tablet 88b77dcdcea8453aa058d542eccb8045 (table test-table [id=75712d63361c4697b060ba953d236471]) successfully deleted
I20260620 02:29:08.931772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 6535
I20260620 02:29:08.997467 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.963767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.97.60:44151)
0620 02:29:08.964117 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.964126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.975199 (+ 11073us) server_negotiation.cc:407] Connection header received
0620 02:29:08.975762 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.975770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.975873 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.975974 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.980817 (+ 4843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.981850 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.984363 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.985216 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.985324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.989657 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.989664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.989669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.989741 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.993643 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.994580 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.994587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.994591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.995079 (+ 488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:08.995532 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:08.995537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:08.995540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:08.995929 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:08.996190 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:08.996812 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 02:29:08.997110 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:29:09.002662 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:08.963279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.0.97.61:46423)
0620 02:29:08.963519 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:08.963527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:08.980290 (+ 16763us) server_negotiation.cc:407] Connection header received
0620 02:29:08.981127 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:08.981134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:08.981239 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:08.981331 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:08.984751 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.986065 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.988801 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:08.989535 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:08.989654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:08.993974 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:08.993983 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:08.993989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:08.994041 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:08.999041 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:09.000300 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:09.000306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:09.000310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:09.000472 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:09.000954 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:09.000959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:09.000963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:09.001365 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:09.001612 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:09.002067 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 02:29:09.002313 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:29:09.723893 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:09.723162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:51997)
0620 02:29:09.723613 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:09.723736 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":218,"threads_started":1}
W20260620 02:29:09.724368 7684 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45531 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:09.728018 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:09.726992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:48239)
0620 02:29:09.727635 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:09.727801 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":405,"threads_started":1}
W20260620 02:29:09.728480 7825 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45531 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:09.736691 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:09.735871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:45347)
0620 02:29:09.736402 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:09.736524 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
W20260620 02:29:09.737210 7544 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45531 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:10.252213 7901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ffa152e2c2a4d54adeaa220f087eee8)
I20260620 02:29:10.252208 7902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ffa152e2c2a4d54adeaa220f087eee8)
I20260620 02:29:10.252526 7902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:10.252534 7901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:10.260426 7901 leader_election.cc:288] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531), 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61:46423)
I20260620 02:29:10.265697 7902 leader_election.cc:288] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531), 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60:44151)
I20260620 02:29:10.269271 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.254319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44554)
0620 02:29:10.268712 (+ 14393us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.268891 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14249,"threads_started":1}
I20260620 02:29:10.269317 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.254337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44568)
0620 02:29:10.268610 (+ 14273us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.268869 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14116,"mutex_wait_us":45,"thread_start_us":13528,"threads_started":1}
W20260620 02:29:10.269915 7151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
W20260620 02:29:10.271129 7345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:10.272538 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.272073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44570)
0620 02:29:10.272272 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.272373 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 02:29:10.272835 7151 leader_election.cc:334] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:10.276965 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.276021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44584)
0620 02:29:10.276206 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.276399 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:29:10.277277 7345 leader_election.cc:334] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:10.278172 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.254356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46423 (local address 127.0.0.1:48068)
0620 02:29:10.269572 (+ 15216us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.269600 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:10.269836 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:10.270846 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:10.270851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:10.270878 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:10.271255 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:10.271271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.273491 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.273498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:10.275613 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:10.275628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.277240 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.277245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:10.277456 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:10.277489 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:29:10.277618 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 02:29:10.277761 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15083,"mutex_wait_us":37,"thread_start_us":14044,"threads_started":1}
I20260620 02:29:10.278379 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.254967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.0.97.60:44151)
0620 02:29:10.269971 (+ 15004us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:10.269979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:10.270002 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:29:10.270074 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:10.270080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:10.270164 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:10.270274 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:10.271175 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.272522 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.275812 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.277080 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.277209 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:10.277393 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:10.277747 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:29:10.277881 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":14863,"thread_start_us":15506,"threads_started":1}
I20260620 02:29:10.278617 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.260245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.0.97.61:46423)
0620 02:29:10.266399 (+ 6154us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:10.266408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:10.270024 (+ 3616us) server_negotiation.cc:407] Connection header received
0620 02:29:10.270122 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:10.270131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:10.270235 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:10.270491 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:10.271495 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.273249 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.275842 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.276989 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.277155 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:10.277270 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:10.277640 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 02:29:10.277771 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":6000,"spinlock_wait_cycles":1280,"thread_start_us":5456,"threads_started":1}
I20260620 02:29:10.278947 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.254340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44151 (local address 127.0.0.1:41838)
0620 02:29:10.264311 (+ 9971us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.264341 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:10.264547 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:10.270507 (+ 5960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:10.270513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:10.270543 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:10.270891 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:10.270906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.272794 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.272800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:10.275611 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:10.275628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:10.277364 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:10.277372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:10.277544 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:10.277594 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 02:29:10.277724 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:29:10.277873 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9815,"mutex_wait_us":21,"thread_start_us":9651,"threads_started":1}
I20260620 02:29:10.279309 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "76105f54940c45e1ad5bb2478c7e4766" is_pre_election: true
I20260620 02:29:10.279657 7177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e5b2f9b5b74d9d84f0c0059dd09ece in term 1.
I20260620 02:29:10.279701 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76105f54940c45e1ad5bb2478c7e4766" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" is_pre_election: true
I20260620 02:29:10.280051 7371 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76105f54940c45e1ad5bb2478c7e4766 in term 1.
I20260620 02:29:10.280206 7348 leader_election.cc:302] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54e5b2f9b5b74d9d84f0c0059dd09ece, 76105f54940c45e1ad5bb2478c7e4766; no voters: 9ffa152e2c2a4d54adeaa220f087eee8
I20260620 02:29:10.280638 7901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:29:10.280584 7152 leader_election.cc:302] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54e5b2f9b5b74d9d84f0c0059dd09ece, 76105f54940c45e1ad5bb2478c7e4766; no voters: 9ffa152e2c2a4d54adeaa220f087eee8
I20260620 02:29:10.280782 7901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ffa152e2c2a4d54adeaa220f087eee8)
I20260620 02:29:10.280889 7901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:29:10.280932 7902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:29:10.281104 7902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ffa152e2c2a4d54adeaa220f087eee8)
I20260620 02:29:10.281220 7902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:29:10.284360 7902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:10.284333 7901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:10.285199 7901 leader_election.cc:288] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 2 election: Requested vote from peers 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531), 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61:46423)
I20260620 02:29:10.285887 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "76105f54940c45e1ad5bb2478c7e4766"
I20260620 02:29:10.286234 7177 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54e5b2f9b5b74d9d84f0c0059dd09ece in current term 2: Already voted for candidate 76105f54940c45e1ad5bb2478c7e4766 in this term.
I20260620 02:29:10.286499 7902 leader_election.cc:288] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [CANDIDATE]: Term 2 election: Requested vote from peers 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531), 54e5b2f9b5b74d9d84f0c0059dd09ece (127.0.97.60:44151)
I20260620 02:29:10.286526 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.285893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44594)
0620 02:29:10.286174 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.286269 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 02:29:10.287638 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76105f54940c45e1ad5bb2478c7e4766" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece"
I20260620 02:29:10.287796 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.287296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44614)
0620 02:29:10.287500 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.287603 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:29:10.287971 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.285408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44596)
0620 02:29:10.287714 (+ 2306us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.287825 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2191}
I20260620 02:29:10.288007 7371 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 76105f54940c45e1ad5bb2478c7e4766 in current term 2: Already voted for candidate 54e5b2f9b5b74d9d84f0c0059dd09ece in this term.
W20260620 02:29:10.288035 7345 leader_election.cc:334] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:10.288166 7345 leader_election.cc:302] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e5b2f9b5b74d9d84f0c0059dd09ece; no voters: 76105f54940c45e1ad5bb2478c7e4766, 9ffa152e2c2a4d54adeaa220f087eee8
I20260620 02:29:10.288450 7901 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 02:29:10.289180 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.288733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:44622)
0620 02:29:10.288901 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.288988 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 02:29:10.289357 7151 leader_election.cc:334] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:10.289508 7151 leader_election.cc:302] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76105f54940c45e1ad5bb2478c7e4766; no voters: 54e5b2f9b5b74d9d84f0c0059dd09ece, 9ffa152e2c2a4d54adeaa220f087eee8
I20260620 02:29:10.289825 7902 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 02:29:10.726022 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.725186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:55421)
0620 02:29:10.725701 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.725799 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260620 02:29:10.730126 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.729259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:41097)
0620 02:29:10.729796 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.729894 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":330,"threads_started":1}
I20260620 02:29:10.738692 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:10.737929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:60293)
0620 02:29:10.738396 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:10.738501 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260620 02:29:11.727825 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:11.727074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:50191)
0620 02:29:11.727556 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:11.727656 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":261,"threads_started":1}
I20260620 02:29:11.731992 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:11.731089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:49381)
0620 02:29:11.731586 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:11.731672 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":290,"threads_started":1}
I20260620 02:29:11.740731 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:11.739772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:41083)
0620 02:29:11.740333 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:11.740464 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":280,"threads_started":1}
I20260620 02:29:12.094553 7917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:12.094856 7917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:12.095593 7917 leader_election.cc:288] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531), 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61:46423)
I20260620 02:29:12.096235 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "76105f54940c45e1ad5bb2478c7e4766" is_pre_election: true
I20260620 02:29:12.096547 7177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e5b2f9b5b74d9d84f0c0059dd09ece in term 2.
I20260620 02:29:12.097033 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.095961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40778)
0620 02:29:12.096580 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.096679 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":353,"threads_started":1}
I20260620 02:29:12.097114 7348 leader_election.cc:302] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e5b2f9b5b74d9d84f0c0059dd09ece, 76105f54940c45e1ad5bb2478c7e4766; no voters:
I20260620 02:29:12.097414 7917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 02:29:12.097595 7917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:12.097721 7917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:29:12.098723 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.098344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40780)
0620 02:29:12.098512 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.098597 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 02:29:12.098991 7345 leader_election.cc:334] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:12.100325 7917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:12.100879 7917 leader_election.cc:288] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 3 election: Requested vote from peers 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531), 76105f54940c45e1ad5bb2478c7e4766 (127.0.97.61:46423)
I20260620 02:29:12.101406 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.100961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40788)
0620 02:29:12.101196 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.101264 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 02:29:12.101402 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "76105f54940c45e1ad5bb2478c7e4766"
I20260620 02:29:12.101637 7177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:29:12.102468 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.102153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40800)
0620 02:29:12.102281 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.102361 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:29:12.102715 7345 leader_election.cc:334] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
I20260620 02:29:12.104116 7177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e5b2f9b5b74d9d84f0c0059dd09ece in term 3.
I20260620 02:29:12.104539 7348 leader_election.cc:302] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54e5b2f9b5b74d9d84f0c0059dd09ece, 76105f54940c45e1ad5bb2478c7e4766; no voters: 9ffa152e2c2a4d54adeaa220f087eee8
I20260620 02:29:12.104877 7917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:29:12.105369 7917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [term 3 LEADER]: Becoming Leader. State: Replica: 54e5b2f9b5b74d9d84f0c0059dd09ece, State: Running, Role: LEADER
I20260620 02:29:12.105806 7917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } }
I20260620 02:29:12.107512 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e5b2f9b5b74d9d84f0c0059dd09ece. Latest consensus state: current_term: 3 leader_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:12.107895 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:12.108676 7922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:12.112342 7922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:12.113821 7922 catalog_manager.cc:1259] Loaded cluster ID: a5a78761f7ee46fbae3a1f0e39d7de34
I20260620 02:29:12.113945 7922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:12.115152 7922 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:12.116014 7922 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Loaded TSK: 0
I20260620 02:29:12.116901 7922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:29:12.141436 7360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:12.149566 7360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:12.154400 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.153821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40814)
0620 02:29:12.154043 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.154181 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 02:29:12.154345 7177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [term 3 FOLLOWER]: Refusing update from remote peer 54e5b2f9b5b74d9d84f0c0059dd09ece: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 02:29:12.155236 7920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 02:29:12.155534 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.155096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40816)
0620 02:29:12.155274 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.155365 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:29:12.155778 7345 consensus_peers.cc:487] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece -> Peer 9ffa152e2c2a4d54adeaa220f087eee8 (127.0.97.62:45531): Couldn't send request to peer 9ffa152e2c2a4d54adeaa220f087eee8. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:29:12.158581 7924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e5b2f9b5b74d9d84f0c0059dd09ece. Latest consensus state: current_term: 3 leader_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:12.158900 7924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:12.160717 7917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:12.161175 7917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:12.161545 7925 mvcc.cc:205] Tried to move back new op lower bound from 7298754773616537600 to 7298754773428867072. Current Snapshot: MvccSnapshot[applied={T|T < 7298754773616537600}]
I20260620 02:29:12.161818 7924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9ffa152e2c2a4d54adeaa220f087eee8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45531 } } peers { permanent_uuid: "76105f54940c45e1ad5bb2478c7e4766" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46423 } attrs { promote: false } } peers { permanent_uuid: "54e5b2f9b5b74d9d84f0c0059dd09ece" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44151 } attrs { promote: false } } }
I20260620 02:29:12.162220 7924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76105f54940c45e1ad5bb2478c7e4766 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:29:12.194617 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.170972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33577 (local address 127.0.0.1:35422)
0620 02:29:12.171199 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.171220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:12.171404 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:12.187210 (+ 15806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:12.187218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:12.187261 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:12.187636 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.187652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.189935 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.189941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.192106 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.192120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.193749 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.193755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.193941 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.193967 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:29:12.194097 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:29:12.194241 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 02:29:12.194608 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.171136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.2:33577)
0620 02:29:12.186737 (+ 15601us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.186747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.186781 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:29:12.186909 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.186917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.187020 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.187161 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:12.188232 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.189752 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.192296 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.193513 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.193681 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.193804 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.194130 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.194264 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15403,"thread_start_us":15187,"threads_started":1}
I20260620 02:29:12.196059 7618 tablet_service.cc:1467] Processing CreateTablet for tablet e2c9904225e341749776c265566e988f (DEFAULT_TABLE table=test-table [id=0ea64e8fd16f45618a0812d4c6f3a865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:12.196623 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c9904225e341749776c265566e988f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:12.197858 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.171974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.0.97.3:44331)
0620 02:29:12.190879 (+ 18905us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.190884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.190901 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:29:12.190968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.190971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.191038 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.191200 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:12.191813 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.192864 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.195429 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.196654 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.197206 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.197324 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.197387 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.197524 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18793,"thread_start_us":18645,"threads_started":1}
I20260620 02:29:12.198033 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.171198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34705 (local address 127.0.0.1:38430)
0620 02:29:12.184608 (+ 13410us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.184636 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:12.185086 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:12.190232 (+ 5146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:12.190237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:12.190271 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:12.190645 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.190656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.192282 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.192287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.195599 (+ 3312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.195619 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.196880 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.196886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.197123 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.197244 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 02:29:12.197336 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:29:12.197559 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13220,"mutex_wait_us":140,"thread_start_us":14209,"threads_started":1}
I20260620 02:29:12.198069 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.171659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44331 (local address 127.0.0.1:40128)
0620 02:29:12.185749 (+ 14090us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.185772 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:12.186027 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:12.191212 (+ 5185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:12.191216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:12.191239 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:12.191632 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.191640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.193080 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.193085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.195240 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.195254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.196934 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.196941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.197211 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.197260 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:29:12.197333 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 02:29:12.197559 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13924,"mutex_wait_us":119,"thread_start_us":13089,"threads_started":1}
I20260620 02:29:12.198665 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.171658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.0.97.1:34705)
0620 02:29:12.189674 (+ 18016us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.189683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.189711 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:12.189860 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.189868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.190013 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.190202 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:12.190904 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.192123 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.195772 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.196632 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.196803 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.196936 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.198276 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.198434 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17780,"thread_start_us":17620,"threads_started":1}
I20260620 02:29:12.199602 7478 tablet_service.cc:1467] Processing CreateTablet for tablet e2c9904225e341749776c265566e988f (DEFAULT_TABLE table=test-table [id=0ea64e8fd16f45618a0812d4c6f3a865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:12.200086 7478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c9904225e341749776c265566e988f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:12.202214 7759 tablet_service.cc:1467] Processing CreateTablet for tablet e2c9904225e341749776c265566e988f (DEFAULT_TABLE table=test-table [id=0ea64e8fd16f45618a0812d4c6f3a865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:12.202711 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c9904225e341749776c265566e988f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:12.203855 7935 tablet_bootstrap.cc:492] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Bootstrap starting.
I20260620 02:29:12.205724 7935 tablet_bootstrap.cc:654] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:12.208585 7935 tablet_bootstrap.cc:492] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: No bootstrap required, opened a new log
I20260620 02:29:12.208850 7935 ts_tablet_manager.cc:1397] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:29:12.209105 7937 tablet_bootstrap.cc:492] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Bootstrap starting.
I20260620 02:29:12.209898 7935 raft_consensus.cc:348] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.211730 7937 tablet_bootstrap.cc:654] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:12.210194 7935 raft_consensus.cc:374] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:12.213146 7935 raft_consensus.cc:729] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb94b6e7171547909fb9523a79f5a0f3, State: Initialized, Role: FOLLOWER
I20260620 02:29:12.213531 7935 consensus_queue.cc:260] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.217900 7935 ts_tablet_manager.cc:1428] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20260620 02:29:12.219337 7938 tablet_bootstrap.cc:492] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Bootstrap starting.
I20260620 02:29:12.219621 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.219083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:41757)
0620 02:29:12.219334 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.219448 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 02:29:12.221117 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.220581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:49033)
0620 02:29:12.220824 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.220920 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 02:29:12.221692 7937 tablet_bootstrap.cc:492] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: No bootstrap required, opened a new log
I20260620 02:29:12.221964 7937 ts_tablet_manager.cc:1397] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260620 02:29:12.222110 7683 heartbeater.cc:503] Master 127.0.97.60:44151 was elected leader, sending a full tablet report...
I20260620 02:29:12.221848 7938 tablet_bootstrap.cc:654] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:12.223111 7937 raft_consensus.cc:348] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.223510 7937 raft_consensus.cc:374] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:12.223659 7937 raft_consensus.cc:729] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c77e3ecfbc452996a662ee3602130e, State: Initialized, Role: FOLLOWER
I20260620 02:29:12.224062 7937 consensus_queue.cc:260] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.225322 7937 ts_tablet_manager.cc:1428] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:29:12.226027 7938 tablet_bootstrap.cc:492] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: No bootstrap required, opened a new log
I20260620 02:29:12.226222 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.225715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:43501)
0620 02:29:12.225953 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.226054 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 02:29:12.226240 7938 ts_tablet_manager.cc:1397] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 02:29:12.227406 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.226874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:50033)
0620 02:29:12.227132 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.227216 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 02:29:12.227319 7938 raft_consensus.cc:348] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.227576 7938 raft_consensus.cc:374] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:12.227576 7543 heartbeater.cc:503] Master 127.0.97.60:44151 was elected leader, sending a full tablet report...
I20260620 02:29:12.227731 7938 raft_consensus.cc:729] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca2b15f70634207a7b438742e69b302, State: Initialized, Role: FOLLOWER
I20260620 02:29:12.228088 7938 consensus_queue.cc:260] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.229146 7938 ts_tablet_manager.cc:1428] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:29:12.230211 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.229727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:48745)
0620 02:29:12.229954 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.230044 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 02:29:12.231374 7824 heartbeater.cc:503] Master 127.0.97.60:44151 was elected leader, sending a full tablet report...
I20260620 02:29:12.231590 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.231129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:51623)
0620 02:29:12.231301 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.231416 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:29:12.259279 7546 tablet.cc:2369] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:12.454288 7940 raft_consensus.cc:482] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:12.454655 7940 raft_consensus.cc:504] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.456446 7940 leader_election.cc:288] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705), 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331)
I20260620 02:29:12.463506 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.456773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34705 (local address 127.0.97.2:34061)
0620 02:29:12.456966 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.457023 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:12.457177 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:12.457681 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:12.457687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:12.457720 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:12.458113 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.458128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.459367 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.459373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.461241 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.461254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.462638 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.462646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.462838 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.462877 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:29:12.462977 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:29:12.463153 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":99}
I20260620 02:29:12.463521 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.457041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34061 (local address 127.0.97.1:34705)
0620 02:29:12.457257 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.457264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.457288 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:29:12.457379 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.457387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.457474 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.457618 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:12.458298 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.459204 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.461410 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.462412 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.462516 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.462596 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.463028 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.463186 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:29:12.463908 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.456928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51323 (local address 127.0.97.3:44331)
0620 02:29:12.457197 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.457204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.457804 (+ 600us) server_negotiation.cc:407] Connection header received
0620 02:29:12.457968 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.457975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.458090 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.458193 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:12.458774 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.459628 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.461832 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.462840 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.462982 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.463093 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.463503 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.463656 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 02:29:12.464062 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.456773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44331 (local address 127.0.97.2:51323)
0620 02:29:12.457683 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.457701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:12.457839 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:12.458229 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:12.458234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:12.458261 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:12.458585 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.458597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.459784 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.459788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.461666 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:12.461680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.463119 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.463126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:12.463329 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.463364 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:29:12.463471 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:29:12.463622 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":237,"thread_start_us":314,"threads_started":1}
I20260620 02:29:12.464892 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c9904225e341749776c265566e988f" candidate_uuid: "cb94b6e7171547909fb9523a79f5a0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca2b15f70634207a7b438742e69b302" is_pre_election: true
I20260620 02:29:12.464954 7498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c9904225e341749776c265566e988f" candidate_uuid: "cb94b6e7171547909fb9523a79f5a0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c77e3ecfbc452996a662ee3602130e" is_pre_election: true
I20260620 02:29:12.465296 7779 raft_consensus.cc:2454] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb94b6e7171547909fb9523a79f5a0f3 in term 0.
I20260620 02:29:12.465626 7498 raft_consensus.cc:2454] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb94b6e7171547909fb9523a79f5a0f3 in term 0.
I20260620 02:29:12.465926 7571 leader_election.cc:302] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ca2b15f70634207a7b438742e69b302, cb94b6e7171547909fb9523a79f5a0f3; no voters:
I20260620 02:29:12.466394 7940 raft_consensus.cc:2781] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:12.466538 7940 raft_consensus.cc:482] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:12.466657 7940 raft_consensus.cc:3037] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:12.470211 7940 raft_consensus.cc:504] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.471035 7940 leader_election.cc:288] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [CANDIDATE]: Term 1 election: Requested vote from peers 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705), 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331)
I20260620 02:29:12.471710 7498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c9904225e341749776c265566e988f" candidate_uuid: "cb94b6e7171547909fb9523a79f5a0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c77e3ecfbc452996a662ee3602130e"
I20260620 02:29:12.471714 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c9904225e341749776c265566e988f" candidate_uuid: "cb94b6e7171547909fb9523a79f5a0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca2b15f70634207a7b438742e69b302"
I20260620 02:29:12.471947 7498 raft_consensus.cc:3037] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:12.471962 7779 raft_consensus.cc:3037] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:12.475303 7779 raft_consensus.cc:2454] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb94b6e7171547909fb9523a79f5a0f3 in term 1.
I20260620 02:29:12.475365 7498 raft_consensus.cc:2454] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb94b6e7171547909fb9523a79f5a0f3 in term 1.
I20260620 02:29:12.475793 7571 leader_election.cc:302] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ca2b15f70634207a7b438742e69b302, cb94b6e7171547909fb9523a79f5a0f3; no voters:
I20260620 02:29:12.476145 7940 raft_consensus.cc:2781] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:12.476647 7940 raft_consensus.cc:686] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 1 LEADER]: Becoming Leader. State: Replica: cb94b6e7171547909fb9523a79f5a0f3, State: Running, Role: LEADER
I20260620 02:29:12.477111 7940 consensus_queue.cc:237] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } }
I20260620 02:29:12.478091 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.477610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:57439)
0620 02:29:12.477779 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.477907 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:29:12.480932 7360 catalog_manager.cc:5495] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "cb94b6e7171547909fb9523a79f5a0f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb94b6e7171547909fb9523a79f5a0f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 } health_report { overall_health: UNKNOWN } } }
I20260620 02:29:12.556727 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.551122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.97.2:33577)
0620 02:29:12.551359 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.551365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.551386 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:29:12.551523 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.551527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.551605 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.551770 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:12.552166 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.552962 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.554376 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.554789 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.555057 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.555572 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:12.556131 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:12.556306 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.556364 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.556446 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:29:12.561275 7779 raft_consensus.cc:1261] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Refusing update from remote peer cb94b6e7171547909fb9523a79f5a0f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:12.561789 7498 raft_consensus.cc:1261] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Refusing update from remote peer cb94b6e7171547909fb9523a79f5a0f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:12.562142 7945 consensus_queue.cc:1035] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca2b15f70634207a7b438742e69b302" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:12.562629 7940 consensus_queue.cc:1035] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c77e3ecfbc452996a662ee3602130e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:12.565701 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.565087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:42513)
0620 02:29:12.565371 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.565505 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260620 02:29:12.565876 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.565247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:56731)
0620 02:29:12.565485 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.565639 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 02:29:12.571203 7948 mvcc.cc:205] Tried to move back new op lower bound from 7298754775283609600 to 7298754774950727680. Current Snapshot: MvccSnapshot[applied={T|T < 7298754775283609600}]
I20260620 02:29:12.573789 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.573212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.1:42187)
0620 02:29:12.573443 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.573579 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 02:29:12.574488 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.573671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.3:38733)
0620 02:29:12.574190 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.574285 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399}
I20260620 02:29:12.575364 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.574838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40824)
0620 02:29:12.575067 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.575186 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 02:29:12.575897 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.575393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.97.2:42917)
0620 02:29:12.575626 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.575716 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 02:29:12.576561 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.576090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40834)
0620 02:29:12.576293 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:12.576390 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:29:12.589226 7360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:12.589558 7360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:12.595712 7360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54e5b2f9b5b74d9d84f0c0059dd09ece: Sending DeleteTablet for 3 replicas of tablet e2c9904225e341749776c265566e988f
I20260620 02:29:12.596745 7618 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c9904225e341749776c265566e988f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:12 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35422
I20260620 02:29:12.596988 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c9904225e341749776c265566e988f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:12 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38430
I20260620 02:29:12.597675 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c9904225e341749776c265566e988f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:12 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40128
I20260620 02:29:12.598695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 7333
I20260620 02:29:12.601523 7958 tablet_replica.cc:330] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: stopping tablet replica
I20260620 02:29:12.601758 7958 raft_consensus.cc:2229] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:12.602099 7958 raft_consensus.cc:2258] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:12.603349 7958 ts_tablet_manager.cc:1905] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:12.611128 7958 ts_tablet_manager.cc:1918] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:12.611443 7958 log.cc:1196] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/e2c9904225e341749776c265566e988f
I20260620 02:29:12.612143 7958 ts_tablet_manager.cc:1939] T e2c9904225e341749776c265566e988f P 78c77e3ecfbc452996a662ee3602130e: Deleting consensus metadata
I20260620 02:29:12.613909 7348 catalog_manager.cc:4841] TS 78c77e3ecfbc452996a662ee3602130e (127.0.97.1:34705): tablet e2c9904225e341749776c265566e988f (table test-table [id=0ea64e8fd16f45618a0812d4c6f3a865]) successfully deleted
I20260620 02:29:12.624119 7959 tablet_replica.cc:330] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: stopping tablet replica
I20260620 02:29:12.624341 7959 raft_consensus.cc:2229] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:12.624650 7959 raft_consensus.cc:2258] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:12.625279 7957 tablet_replica.cc:330] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: stopping tablet replica
I20260620 02:29:12.625805 7957 raft_consensus.cc:2229] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:12.626561 7957 raft_consensus.cc:2258] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:12.628367 7957 ts_tablet_manager.cc:1905] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:12.634958 7959 ts_tablet_manager.cc:1905] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:12.635715 7957 ts_tablet_manager.cc:1918] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:12.635936 7957 log.cc:1196] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/e2c9904225e341749776c265566e988f
I20260620 02:29:12.636490 7957 ts_tablet_manager.cc:1939] T e2c9904225e341749776c265566e988f P cb94b6e7171547909fb9523a79f5a0f3: Deleting consensus metadata
I20260620 02:29:12.637943 7345 catalog_manager.cc:4841] TS cb94b6e7171547909fb9523a79f5a0f3 (127.0.97.2:33577): tablet e2c9904225e341749776c265566e988f (table test-table [id=0ea64e8fd16f45618a0812d4c6f3a865]) successfully deleted
I20260620 02:29:12.641698 7959 ts_tablet_manager.cc:1918] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:12.642218 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.599820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.0.97.1:34705)
0620 02:29:12.599999 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.600007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.623335 (+ 23328us) server_negotiation.cc:407] Connection header received
0620 02:29:12.623509 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.623515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.623617 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.623737 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:12.625926 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.627175 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.629204 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.629858 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.629963 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.633837 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:12.633859 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:12.633880 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:12.633940 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:12.638670 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:12.639686 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:12.639695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:12.639699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:12.639868 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:12.640357 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:12.640363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:12.640367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:12.640835 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:12.641208 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.641599 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.641861 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:29:12.643278 7538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:12.644349 7959 log.cc:1196] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/e2c9904225e341749776c265566e988f
I20260620 02:29:12.646209 7959 ts_tablet_manager.cc:1939] T e2c9904225e341749776c265566e988f P 6ca2b15f70634207a7b438742e69b302: Deleting consensus metadata
I20260620 02:29:12.652926 7345 catalog_manager.cc:4841] TS 6ca2b15f70634207a7b438742e69b302 (127.0.97.3:44331): tablet e2c9904225e341749776c265566e988f (table test-table [id=0ea64e8fd16f45618a0812d4c6f3a865]) successfully deleted
I20260620 02:29:12.796257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 7333
I20260620 02:29:12.827894 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 7555
I20260620 02:29:12.842507 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.828710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.97.2:33577)
0620 02:29:12.828882 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.828889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.828949 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:29:12.829208 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.829214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.829290 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.829425 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:12.830733 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.831602 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.833064 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.833537 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.833679 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.836335 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:12.836347 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:12.836362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:12.836410 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:12.839469 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:12.840196 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:12.840210 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:12.840212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:12.840382 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:12.840835 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:12.840840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:12.840843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:12.841275 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:12.841604 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:12.841896 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:29:12.842130 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:29:12.843304 7678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:12.968158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 7555
I20260620 02:29:12.989043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 7696
I20260620 02:29:13.003679 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:12.989928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.0.97.3:44331)
0620 02:29:12.990430 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:12.990435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:12.990452 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:29:12.990505 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:12.990509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:12.990569 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:12.990661 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:12.992120 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.993334 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.994566 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:12.995041 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:12.995141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:12.997897 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:12.997911 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:12.997927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:12.997964 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:13.001117 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:13.001794 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:13.001802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:13.001805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:13.001939 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:13.002264 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:13.002267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:13.002269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:13.002623 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:13.002895 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:13.003098 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 02:29:13.003287 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
I20260620 02:29:13.004511 7819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:13.108130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 7696
I20260620 02:29:13.126323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 7030
I20260620 02:29:13.127627 7197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:13.127748 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:13.126997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40840)
0620 02:29:13.127501 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:13.127585 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260620 02:29:13.129313 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:13.128840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45531 (local address 127.0.0.1:40846)
0620 02:29:13.129047 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:13.129146 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 02:29:13.250794 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 7030
I20260620 02:29:13.272761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 7131
I20260620 02:29:13.273962 7391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:13.405543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 7131
2026-06-20T02:29:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12866 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:29:13 dist-test-slave-r706 krb5kdc[7971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:29:13 dist-test-slave-r706 krb5kdc[7971](info): set up 2 sockets
Jun 20 02:29:13 dist-test-slave-r706 krb5kdc[7971](info): commencing operation
krb5kdc: starting...
W20260620 02:29:14.550230 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:29:14 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:29:14.793494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40939
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:15.135866 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:15.136185 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:15.136368 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:15.144009 7984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:15.144125 7984 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:29:15.144213 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:15.144335 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:15.144431 7984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:15.144544 7984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:15.154976 7984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:15.156347 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:15.157819 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:15.170917 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:15.170941 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:15.171226 7984 server_base.cc:1034] running on GCE node
W20260620 02:29:15.170912 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:15.172693 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:15.174233 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:15.175442 7984 hybrid_clock.cc:648] HybridClock initialized: now 1781922555175343 us; error 68 us; skew 500 ppm
I20260620 02:29:15.175843 7984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:15 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:15.179181 7984 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:29:15.180677 7984 webserver.cc:460] Webserver started at http://127.0.97.62:37513/ using document root <none> and password file <none>
I20260620 02:29:15.181448 7984 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:15.181540 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:15.181805 7984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:15.184767 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "45f5a5a2c7c54422a885e130a5a4e9c7"
format_stamp: "Formatted at 2026-06-20 02:29:15 on dist-test-slave-r706"
server_key: "89e097f8d403b75b1c3866a310d51cb0"
server_key_iv: "2f9b9d524c0ddd8988694b122ca0995d"
server_key_version: "encryptionkey@0"
I20260620 02:29:15.185745 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "45f5a5a2c7c54422a885e130a5a4e9c7"
format_stamp: "Formatted at 2026-06-20 02:29:15 on dist-test-slave-r706"
server_key: "89e097f8d403b75b1c3866a310d51cb0"
server_key_iv: "2f9b9d524c0ddd8988694b122ca0995d"
server_key_version: "encryptionkey@0"
I20260620 02:29:15.190855 7984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:29:15.194460 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:15.195966 7984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:29:15.196173 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "45f5a5a2c7c54422a885e130a5a4e9c7"
format_stamp: "Formatted at 2026-06-20 02:29:15 on dist-test-slave-r706"
server_key: "89e097f8d403b75b1c3866a310d51cb0"
server_key_iv: "2f9b9d524c0ddd8988694b122ca0995d"
server_key_version: "encryptionkey@0"
I20260620 02:29:15.196406 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:15.221899 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:15.228106 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:15.228358 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:15.252302 7984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40939
I20260620 02:29:15.252277 8060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40939 every 8 connection(s)
I20260620 02:29:15.253801 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:15.257309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 7984
I20260620 02:29:15.257515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:29:15.257934 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:15.258378 388 external_mini_cluster.cc:1428] Setting key a3cabdd2fe299d7136124c893aff369a
I20260620 02:29:15.273204 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Bootstrap starting.
Jun 20 02:29:15 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:15.276495 8061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:15.277915 8061 log.cc:824] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:15.281520 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: No bootstrap required, opened a new log
I20260620 02:29:15.285471 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:15.260795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.0.97.62:40939)
0620 02:29:15.261827 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:15.261844 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:15.261895 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:29:15.265034 (+ 3139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:15.265066 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:15.265653 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:15.266134 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:15.267606 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.269688 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.270997 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.272137 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.272322 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:15.276546 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:15.276557 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:15.276578 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:15.276629 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:15.280598 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:15.281417 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:15.281425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:15.281433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:15.281753 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:15.282126 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:15.282131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:15.282134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:15.282803 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:15.283111 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:15.283475 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 02:29:15.283994 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":494,"threads_started":1}
I20260620 02:29:15.291078 8061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40939 } }
I20260620 02:29:15.291431 8061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:15.291515 8061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f5a5a2c7c54422a885e130a5a4e9c7, State: Initialized, Role: FOLLOWER
I20260620 02:29:15.292073 8061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40939 } }
I20260620 02:29:15.292273 8061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:15.292361 8061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:15.292477 8061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:15.295055 8061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40939 } }
I20260620 02:29:15.295578 8061 leader_election.cc:302] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f5a5a2c7c54422a885e130a5a4e9c7; no voters:
I20260620 02:29:15.296208 8061 leader_election.cc:288] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:29:15.296416 8066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:15.297480 8066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [term 1 LEADER]: Becoming Leader. State: Replica: 45f5a5a2c7c54422a885e130a5a4e9c7, State: Running, Role: LEADER
I20260620 02:29:15.298041 8066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40939 } }
I20260620 02:29:15.298524 8061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:15.302028 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f5a5a2c7c54422a885e130a5a4e9c7. Latest consensus state: current_term: 1 leader_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40939 } } }
I20260620 02:29:15.302529 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:15.309290 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f5a5a2c7c54422a885e130a5a4e9c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40939 } } }
I20260620 02:29:15.309679 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:15.310760 8075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:15.315104 8075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:15.323952 8075 catalog_manager.cc:1347] Generated new cluster ID: 662deafc7b814eb7a784715350588095
I20260620 02:29:15.324096 8075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:15.337337 8075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:29:15.338673 8075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:15.346707 8075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Generated new TSK 0
I20260620 02:29:15.347733 8075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:29:15.408354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40939
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:15.746600 8087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:15.746942 8087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:15.747059 8087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:15.755029 8087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:15.755168 8087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:15.755264 8087 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:29:15.755354 8087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:15.765610 8087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:15.766893 8087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:15.768210 8087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:15.782550 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:15.782482 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:15.782521 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:15.783389 8087 server_base.cc:1034] running on GCE node
I20260620 02:29:15.784070 8087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:15.785198 8087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:15.786450 8087 hybrid_clock.cc:648] HybridClock initialized: now 1781922555786380 us; error 36 us; skew 500 ppm
I20260620 02:29:15.786813 8087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:15 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:15.790462 8087 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:29:15.792212 8087 webserver.cc:460] Webserver started at http://127.0.97.1:33775/ using document root <none> and password file <none>
I20260620 02:29:15.793051 8087 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:15.793210 8087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:15.793573 8087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:15.796779 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
format_stamp: "Formatted at 2026-06-20 02:29:15 on dist-test-slave-r706"
server_key: "d8029423dcb2d726ab176a9616636c7b"
server_key_iv: "a1fba7a44be23844fe248abaae9eae17"
server_key_version: "encryptionkey@0"
I20260620 02:29:15.797746 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
format_stamp: "Formatted at 2026-06-20 02:29:15 on dist-test-slave-r706"
server_key: "d8029423dcb2d726ab176a9616636c7b"
server_key_iv: "a1fba7a44be23844fe248abaae9eae17"
server_key_version: "encryptionkey@0"
I20260620 02:29:15.803587 8087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:29:15.807343 8102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:15.808802 8087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:29:15.809078 8087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
format_stamp: "Formatted at 2026-06-20 02:29:15 on dist-test-slave-r706"
server_key: "d8029423dcb2d726ab176a9616636c7b"
server_key_iv: "a1fba7a44be23844fe248abaae9eae17"
server_key_version: "encryptionkey@0"
I20260620 02:29:15.809340 8087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:15.831916 8087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:15.836304 8087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:15.836624 8087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:15.839617 8087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:15.839730 8087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:15.839870 8087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:15.840018 8087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 02:29:15 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:15.865254 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:15.840081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53543 (local address 127.0.97.62:40939)
0620 02:29:15.840918 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:15.840927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:15.842008 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 02:29:15.844006 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:15.844012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:15.844107 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:15.844242 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:15.846840 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.847949 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.850879 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.851586 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.851736 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:15.857236 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:15.857248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:15.857252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:15.857287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:15.860650 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:15.861599 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:15.861607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:15.861613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:15.861791 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:15.862527 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:15.862533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:15.862537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:15.862964 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:15.863148 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:15.864428 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0620 02:29:15.864711 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":595,"threads_started":1}
I20260620 02:29:15.866991 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:15.840781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.97.1:53543)
0620 02:29:15.841821 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:15.841879 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:15.843714 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:15.844484 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:15.844492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:15.845381 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:15.846590 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:15.846625 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.848163 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.848175 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:15.850713 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:15.850726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.851852 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.851861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:15.852104 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:15.853061 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:15.853109 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:15.856906 (+ 3797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:15.860798 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:15.860807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:15.860828 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:15.861416 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:15.861973 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:15.861981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:15.861985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:15.862346 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:15.863245 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:15.863258 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:15.863864 (+ 606us) client_negotiation.cc:769] Sending connection context
0620 02:29:15.864274 (+ 410us) client_negotiation.cc:240] Negotiation successful
0620 02:29:15.865026 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":448,"threads_started":1}
I20260620 02:29:15.888551 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:15.873916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42841 (local address 127.0.97.62:40939)
0620 02:29:15.874088 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:15.874092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:15.874849 (+ 757us) server_negotiation.cc:407] Connection header received
0620 02:29:15.875100 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:15.875104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:15.875169 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:15.875286 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:15.876272 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.877282 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.879460 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.879875 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.880035 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:15.882741 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:15.882746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:15.882755 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:15.882790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:15.885788 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:15.886504 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:15.886511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:15.886515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:15.886677 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:15.887047 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:15.887054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:15.887057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:15.887357 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:15.887524 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:15.887934 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 02:29:15.888173 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:29:15.888633 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:15.874038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.97.1:42841)
0620 02:29:15.874721 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:15.874737 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:15.874924 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:15.875311 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:15.875315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:15.875814 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:15.876113 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:15.876122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.877464 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.877470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:15.879292 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:15.879306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:15.880092 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:15.880099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:15.880284 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:15.881117 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:15.881137 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:15.882510 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:15.885934 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:15.885942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:15.885947 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:15.886373 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:15.886801 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:15.886806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:15.886809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:15.886938 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:15.887486 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:15.887491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:15.887680 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 02:29:15.887841 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 02:29:15.888186 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":398,"threads_started":1}
I20260620 02:29:15.896926 8087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44453
I20260620 02:29:15.897074 8222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44453 every 8 connection(s)
I20260620 02:29:15.898339 8087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:15.900784 8223 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40939
I20260620 02:29:15.901058 8223 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:15.901702 8223 heartbeater.cc:511] Master 127.0.97.62:40939 requested a full tablet report, sending...
I20260620 02:29:15.902689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 8087
I20260620 02:29:15.902930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:29:15.903793 388 external_mini_cluster.cc:1428] Setting key f228be09f698fd0c813d40bc3c494651
I20260620 02:29:15.904192 8016 ts_manager.cc:194] Registered new tserver with Master: 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453)
I20260620 02:29:15.906329 8016 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53543
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:29:15.956830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40939
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:16.286592 8227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:16.286936 8227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:16.287113 8227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:16.294679 8227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:16.294853 8227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:16.295025 8227 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:29:16.295186 8227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:16.305660 8227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:16.306999 8227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:16.308377 8227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:16.322263 8232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:16.322187 8233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:16.322149 8235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:16.323318 8227 server_base.cc:1034] running on GCE node
I20260620 02:29:16.323992 8227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:16.324981 8227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:16.326298 8227 hybrid_clock.cc:648] HybridClock initialized: now 1781922556326236 us; error 43 us; skew 500 ppm
I20260620 02:29:16.326645 8227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:16 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:16.334961 8227 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:29:16.336639 8227 webserver.cc:460] Webserver started at http://127.0.97.2:35831/ using document root <none> and password file <none>
I20260620 02:29:16.337473 8227 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:16.337625 8227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:16.338016 8227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:16.341065 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519"
format_stamp: "Formatted at 2026-06-20 02:29:16 on dist-test-slave-r706"
server_key: "8bd9a6a7d02fbf351ca7305eca05f985"
server_key_iv: "ca29bdc7575d421bda967b72dc23cc29"
server_key_version: "encryptionkey@0"
I20260620 02:29:16.341876 8227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519"
format_stamp: "Formatted at 2026-06-20 02:29:16 on dist-test-slave-r706"
server_key: "8bd9a6a7d02fbf351ca7305eca05f985"
server_key_iv: "ca29bdc7575d421bda967b72dc23cc29"
server_key_version: "encryptionkey@0"
I20260620 02:29:16.347313 8227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 02:29:16.350965 8242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:16.352454 8227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 02:29:16.352607 8227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519"
format_stamp: "Formatted at 2026-06-20 02:29:16 on dist-test-slave-r706"
server_key: "8bd9a6a7d02fbf351ca7305eca05f985"
server_key_iv: "ca29bdc7575d421bda967b72dc23cc29"
server_key_version: "encryptionkey@0"
I20260620 02:29:16.352872 8227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:16.365723 8227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:16.370709 8227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:16.371011 8227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:16.373814 8227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:16.373907 8227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:16.374095 8227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:16.374181 8227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:29:16 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:16.403107 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.374345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59821 (local address 127.0.97.62:40939)
0620 02:29:16.374549 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:16.374558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:16.376111 (+ 1553us) server_negotiation.cc:407] Connection header received
0620 02:29:16.379191 (+ 3080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:16.379196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:16.379269 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:16.379401 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:16.382183 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.383415 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.386369 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.386848 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.386997 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.392990 (+ 5993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:16.393029 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:16.393036 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:16.393086 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:16.398930 (+ 5844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.400314 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.400320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.400324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.400530 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.401155 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.401162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.401166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.401486 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:16.401667 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:16.402485 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 02:29:16.402724 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:29:16.404194 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.374963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.97.2:59821)
0620 02:29:16.375901 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:16.375963 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:16.378849 (+ 2886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:16.379673 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:16.379682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:16.380547 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:16.381780 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.381814 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.383668 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.383675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.386183 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.386196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.387265 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.387278 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.387747 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.388664 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:16.388705 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:16.392523 (+ 3818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:16.399124 (+ 6601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.399138 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.399162 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.400095 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.400682 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.400688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.400694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.400951 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.401643 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:16.401651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:16.402091 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 02:29:16.402367 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 02:29:16.402924 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":349,"threads_started":1}
I20260620 02:29:16.427752 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.410690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54549 (local address 127.0.97.62:40939)
0620 02:29:16.410944 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:16.410953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:16.411564 (+ 611us) server_negotiation.cc:407] Connection header received
0620 02:29:16.411761 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:16.411764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:16.411825 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:16.411907 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:16.413768 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.414647 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.417382 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.418208 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.418378 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.420824 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:16.420834 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:16.420842 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:16.420892 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:16.424249 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.425395 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.425401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.425405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.425586 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.426070 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.426075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.426079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.426599 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:16.426762 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:16.427181 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 02:29:16.427413 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:29:16.427888 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.410594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.97.2:54549)
0620 02:29:16.411369 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:16.411460 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:16.411636 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:16.411980 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:16.411987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:16.412795 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:16.413546 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.413571 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.414788 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.414794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.417107 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.417126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.418439 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.418449 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.418682 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.419436 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:16.419453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:16.420593 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:16.424395 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.424400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.424405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.425239 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.425741 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.425748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.425758 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.425924 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.426728 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:16.426733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:16.426927 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 02:29:16.427072 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 02:29:16.427414 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":408,"threads_started":1}
I20260620 02:29:16.433596 8227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38733
I20260620 02:29:16.433634 8361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38733 every 8 connection(s)
I20260620 02:29:16.434854 8227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:16.437126 8362 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40939
I20260620 02:29:16.437368 8362 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:16.438035 8362 heartbeater.cc:511] Master 127.0.97.62:40939 requested a full tablet report, sending...
I20260620 02:29:16.439496 8016 ts_manager.cc:194] Registered new tserver with Master: 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733)
I20260620 02:29:16.440472 8016 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59821
I20260620 02:29:16.440833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 8227
I20260620 02:29:16.441082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:29:16.441570 388 external_mini_cluster.cc:1428] Setting key a1f38c8dfa05951f368d1a74e02fd3af
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:29:16.498281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40939
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:16.840529 8366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:16.840819 8366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:16.841085 8366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:16.848819 8366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:16.848938 8366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:16.849109 8366 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:29:16.849249 8366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:16.859687 8366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:16.861086 8366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:16.862668 8366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:16.876506 8371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:16.876541 8374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:16.876609 8372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:16.877450 8366 server_base.cc:1034] running on GCE node
I20260620 02:29:16.878058 8366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:16.879045 8366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:16.880313 8366 hybrid_clock.cc:648] HybridClock initialized: now 1781922556880252 us; error 46 us; skew 500 ppm
I20260620 02:29:16.880688 8366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:16 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:16.884151 8366 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:29:16.885789 8366 webserver.cc:460] Webserver started at http://127.0.97.3:45933/ using document root <none> and password file <none>
I20260620 02:29:16.886507 8366 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:16.886600 8366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:16.886879 8366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:16.889966 8366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
format_stamp: "Formatted at 2026-06-20 02:29:16 on dist-test-slave-r706"
server_key: "a14bae8a47a1d05120bb151fdaf68f2a"
server_key_iv: "0f9aa9e55450a7dd7bd4b1ccd87e441d"
server_key_version: "encryptionkey@0"
I20260620 02:29:16.890760 8366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
format_stamp: "Formatted at 2026-06-20 02:29:16 on dist-test-slave-r706"
server_key: "a14bae8a47a1d05120bb151fdaf68f2a"
server_key_iv: "0f9aa9e55450a7dd7bd4b1ccd87e441d"
server_key_version: "encryptionkey@0"
I20260620 02:29:16.896296 8366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:29:16.901176 8381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:16.902891 8366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:29:16.903041 8366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
format_stamp: "Formatted at 2026-06-20 02:29:16 on dist-test-slave-r706"
server_key: "a14bae8a47a1d05120bb151fdaf68f2a"
server_key_iv: "0f9aa9e55450a7dd7bd4b1ccd87e441d"
server_key_version: "encryptionkey@0"
I20260620 02:29:16.903247 8366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:16.909878 8223 heartbeater.cc:503] Master 127.0.97.62:40939 was elected leader, sending a full tablet report...
I20260620 02:29:16.926915 8366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:16.931339 8366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:16.931658 8366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:16.934667 8366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:16.934791 8366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:16.934988 8366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:16.935140 8366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 02:29:16 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:16.965678 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.935197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56179 (local address 127.0.97.62:40939)
0620 02:29:16.935709 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:16.935715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:16.936978 (+ 1263us) server_negotiation.cc:407] Connection header received
0620 02:29:16.939537 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:16.939544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:16.939667 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:16.939835 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:16.942591 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.944148 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.947006 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.947706 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.947870 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.954175 (+ 6305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:16.954185 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:16.954192 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:16.954244 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:16.958364 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.960601 (+ 2237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.960611 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.960618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.960839 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.961581 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.961590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.961596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.962133 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:16.962306 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:16.965225 (+ 2919us) server_negotiation.cc:299] Negotiation successful
0620 02:29:16.965406 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
I20260620 02:29:16.966176 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.935857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.97.3:56179)
0620 02:29:16.936767 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:16.936847 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:16.939078 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:16.940189 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:16.940200 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:16.941224 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:16.942293 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.942320 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.944432 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.944440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.946797 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.946819 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.948021 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.948029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.948405 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.949444 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:16.949471 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:16.953746 (+ 4275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:16.958603 (+ 4857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.958647 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.959172 (+ 525us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.960284 (+ 1112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.961046 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.961055 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.961061 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.961388 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.962391 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:16.962410 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:16.963079 (+ 669us) client_negotiation.cc:769] Sending connection context
0620 02:29:16.963464 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 02:29:16.964226 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":364,"threads_started":1}
I20260620 02:29:16.988360 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.972293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39201 (local address 127.0.97.62:40939)
0620 02:29:16.972547 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:16.972556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:16.973272 (+ 716us) server_negotiation.cc:407] Connection header received
0620 02:29:16.973534 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:16.973540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:16.973616 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:16.973733 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:16.975423 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.976721 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.978328 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.979232 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.979407 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.981643 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:16.981649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:16.981652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:16.981686 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:16.985374 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.986343 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.986348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.986352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.986489 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:16.986892 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:16.986896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:16.986904 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:16.987178 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:16.987343 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:16.987755 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 02:29:16.988002 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 02:29:16.988502 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:16.972390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.97.3:39201)
0620 02:29:16.973115 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:16.973140 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:16.973321 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:16.973943 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:16.973950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:16.974729 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:16.975223 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.975242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.976957 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.976965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.978142 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:16.978155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:16.979484 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:16.979489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:16.979620 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:16.980405 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:16.980424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:16.981411 (+ 987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:16.985518 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.985524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.985528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.986194 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.986606 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:16.986613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:16.986616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:16.986757 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:16.987303 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:16.987308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:16.987505 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 02:29:16.987650 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 02:29:16.988001 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":386,"threads_started":1}
I20260620 02:29:17.000473 8366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39025
I20260620 02:29:17.000512 8501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39025 every 8 connection(s)
I20260620 02:29:17.001910 8366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:17.004081 8502 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40939
I20260620 02:29:17.004316 8502 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:17.004906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 8366
I20260620 02:29:17.005137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:29:17.005255 8502 heartbeater.cc:511] Master 127.0.97.62:40939 requested a full tablet report, sending...
I20260620 02:29:17.005782 388 external_mini_cluster.cc:1428] Setting key 8b6184a06d8bfa7b0a913f35f0dca500
I20260620 02:29:17.007057 8016 ts_manager.cc:194] Registered new tserver with Master: cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.008265 8016 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56179
I20260620 02:29:17.009577 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:29:17.066622 8389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:17.054994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.97.62:40939)
0620 02:29:17.055246 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.055253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.055421 (+ 168us) server_negotiation.cc:407] Connection header received
0620 02:29:17.055656 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.055666 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.055732 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.055854 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:17.057510 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.058298 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.059676 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.060088 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.060239 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.066253 (+ 6014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48430: BlockingRecv error: recv got EOF from 127.0.0.1:48430 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:17.084656 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.069998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.0.97.62:40939)
0620 02:29:17.070193 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.070202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.070605 (+ 403us) server_negotiation.cc:407] Connection header received
0620 02:29:17.070821 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.070826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.070899 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.071008 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:17.072148 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.072861 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.074135 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.074594 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.074735 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.078728 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:17.078733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:17.078737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:17.078771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:17.081840 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:17.082590 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:17.082596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:17.082599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:17.082743 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:17.083147 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:17.083152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:17.083156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:17.083450 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:17.083748 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.084097 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.084341 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:29:17.089579 8015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:17.096547 8015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:17.123551 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.109369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44453 (local address 127.0.0.1:49546)
0620 02:29:17.111072 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.111135 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.111345 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.114633 (+ 3288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.114641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.114678 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.115107 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.115126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.117243 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.117249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.120613 (+ 3364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.120629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.122635 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.122670 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.122845 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.122882 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.123007 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.123149 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1358,"mutex_wait_us":325,"thread_start_us":1179,"threads_started":1}
I20260620 02:29:17.123557 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.109605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.0.97.1:44453)
0620 02:29:17.113716 (+ 4111us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.113724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.113754 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:29:17.113875 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.113887 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.114176 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.115317 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.115495 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.117064 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.120814 (+ 3750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.122281 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.122530 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.122750 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.123065 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.123244 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3955,"thread_start_us":3801,"threads_started":1}
I20260620 02:29:17.123754 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.110341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.0.97.2:38733)
0620 02:29:17.112692 (+ 2351us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.112702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.113270 (+ 568us) server_negotiation.cc:407] Connection header received
0620 02:29:17.114601 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.114608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.114827 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.115037 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.115771 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.117581 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.120033 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.121154 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.121290 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.121501 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.123262 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.123443 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2180,"thread_start_us":2031,"threads_started":1}
I20260620 02:29:17.123162 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.109718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38733 (local address 127.0.0.1:51398)
0620 02:29:17.113158 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.113178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.114418 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.115039 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.115045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.115076 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.115570 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.115582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.117756 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.117761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.119833 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.119847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.121777 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.121788 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.122208 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.122313 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.122499 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.122719 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3292,"mutex_wait_us":52,"thread_start_us":4563,"threads_started":1}
I20260620 02:29:17.124642 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.112475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39025 (local address 127.0.0.1:52202)
0620 02:29:17.114108 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.114130 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.114257 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.115298 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.115303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.115329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.115773 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.115783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.117408 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.117414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.120613 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.120629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.123802 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.123808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.123972 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.124006 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.124108 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.124241 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":351,"thread_start_us":1350,"threads_started":1}
I20260620 02:29:17.124725 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.113692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.97.3:39025)
0620 02:29:17.114438 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.114443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.114505 (+ 62us) server_negotiation.cc:407] Connection header received
0620 02:29:17.114599 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.114606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.114827 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.115037 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.115989 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.117239 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.120800 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.123267 (+ 2467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.123399 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.123647 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.124198 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.124393 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":749,"threads_started":1}
I20260620 02:29:17.127914 8292 tablet_service.cc:1467] Processing CreateTablet for tablet bdba1e7a07024e86a9a124de3ad9c8c9 (DEFAULT_TABLE table=test-table [id=2fab63cacd544b2caad37213e3da1a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.128062 8153 tablet_service.cc:1467] Processing CreateTablet for tablet bdba1e7a07024e86a9a124de3ad9c8c9 (DEFAULT_TABLE table=test-table [id=2fab63cacd544b2caad37213e3da1a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.128067 8432 tablet_service.cc:1467] Processing CreateTablet for tablet bdba1e7a07024e86a9a124de3ad9c8c9 (DEFAULT_TABLE table=test-table [id=2fab63cacd544b2caad37213e3da1a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.129380 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdba1e7a07024e86a9a124de3ad9c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.129431 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdba1e7a07024e86a9a124de3ad9c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.129431 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdba1e7a07024e86a9a124de3ad9c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.143544 8526 tablet_bootstrap.cc:492] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Bootstrap starting.
I20260620 02:29:17.144467 8528 tablet_bootstrap.cc:492] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Bootstrap starting.
I20260620 02:29:17.145292 8527 tablet_bootstrap.cc:492] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Bootstrap starting.
I20260620 02:29:17.147413 8526 tablet_bootstrap.cc:654] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.148347 8528 tablet_bootstrap.cc:654] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.148761 8527 tablet_bootstrap.cc:654] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.148789 8526 log.cc:824] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:17.149580 8528 log.cc:824] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:17.150045 8527 log.cc:824] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:17.152305 8526 tablet_bootstrap.cc:492] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: No bootstrap required, opened a new log
I20260620 02:29:17.152650 8528 tablet_bootstrap.cc:492] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: No bootstrap required, opened a new log
I20260620 02:29:17.152683 8526 ts_tablet_manager.cc:1397] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 02:29:17.152974 8528 ts_tablet_manager.cc:1397] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 02:29:17.153254 8527 tablet_bootstrap.cc:492] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: No bootstrap required, opened a new log
I20260620 02:29:17.153517 8527 ts_tablet_manager.cc:1397] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:29:17.160475 8528 raft_consensus.cc:348] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.160888 8528 raft_consensus.cc:374] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.161057 8528 raft_consensus.cc:729] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7bcd3f16d841ae96b3dfc05fe8d5ac, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.161715 8528 consensus_queue.cc:260] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.163683 8526 raft_consensus.cc:348] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.164193 8526 raft_consensus.cc:374] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.164330 8526 raft_consensus.cc:729] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.164129 8527 raft_consensus.cc:348] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.164579 8502 heartbeater.cc:503] Master 127.0.97.62:40939 was elected leader, sending a full tablet report...
I20260620 02:29:17.164611 8527 raft_consensus.cc:374] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.164719 8527 raft_consensus.cc:729] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac14a0cc8f5493a8ced732c66eb78da, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.165184 8526 consensus_queue.cc:260] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.165354 8528 ts_tablet_manager.cc:1428] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 02:29:17.165441 8527 consensus_queue.cc:260] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.166594 8362 heartbeater.cc:503] Master 127.0.97.62:40939 was elected leader, sending a full tablet report...
I20260620 02:29:17.167047 8526 ts_tablet_manager.cc:1428] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 02:29:17.167500 8527 ts_tablet_manager.cc:1428] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260620 02:29:17.172381 8533 raft_consensus.cc:482] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:17.172658 8533 raft_consensus.cc:504] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.174500 8533 leader_election.cc:288] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.181401 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.174930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51503 (local address 127.0.97.1:44453)
0620 02:29:17.175147 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.175154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.175541 (+ 387us) server_negotiation.cc:407] Connection header received
0620 02:29:17.175822 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.175829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.175930 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.176000 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.176943 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.177696 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.179599 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.180363 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.180542 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.180651 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.180992 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.181139 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:29:17.181581 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.175518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53183 (local address 127.0.97.3:39025)
0620 02:29:17.175726 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.175734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.175753 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:29:17.176034 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.176041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.176114 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.176211 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.177064 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.177754 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.179802 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.180580 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.180733 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.180846 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.181200 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.181324 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:29:17.181720 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.174924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39025 (local address 127.0.97.2:53183)
0620 02:29:17.175675 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.175693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.175809 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.176253 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.176257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.176287 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.176847 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.176866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.177901 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.177906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.179647 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.179659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.180804 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.180810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.180978 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.181058 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.181176 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.181307 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":21,"thread_start_us":423,"threads_started":1}
I20260620 02:29:17.181572 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.174724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44453 (local address 127.0.97.2:51503)
0620 02:29:17.175434 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.175457 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.175654 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.176202 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.176206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.176230 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.176732 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.176745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.177835 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.177842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.179445 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.179457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.180604 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.180611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.180800 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.180849 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.180969 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.181134 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":53,"thread_start_us":432,"threads_started":1}
I20260620 02:29:17.182686 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdba1e7a07024e86a9a124de3ad9c8c9" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" is_pre_election: true
I20260620 02:29:17.182868 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdba1e7a07024e86a9a124de3ad9c8c9" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" is_pre_election: true
I20260620 02:29:17.183276 8173 raft_consensus.cc:2454] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:17.183414 8457 raft_consensus.cc:2454] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:17.184031 8245 leader_election.cc:302] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, 6ac14a0cc8f5493a8ced732c66eb78da; no voters:
I20260620 02:29:17.184484 8533 raft_consensus.cc:2781] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:17.184628 8533 raft_consensus.cc:482] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:17.184729 8533 raft_consensus.cc:3037] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.187078 8533 raft_consensus.cc:504] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
W20260620 02:29:17.187273 8363 tablet.cc:2369] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:17.187776 8533 leader_election.cc:288] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Requested vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.188318 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdba1e7a07024e86a9a124de3ad9c8c9" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
I20260620 02:29:17.188524 8457 raft_consensus.cc:3037] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.189234 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdba1e7a07024e86a9a124de3ad9c8c9" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
I20260620 02:29:17.189452 8173 raft_consensus.cc:3037] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.190953 8457 raft_consensus.cc:2454] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:17.191412 8246 leader_election.cc:302] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, cf7bcd3f16d841ae96b3dfc05fe8d5ac; no voters:
I20260620 02:29:17.191820 8533 raft_consensus.cc:2781] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:17.192004 8173 raft_consensus.cc:2454] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:17.192551 8533 raft_consensus.cc:686] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Becoming Leader. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Running, Role: LEADER
I20260620 02:29:17.193149 8533 consensus_queue.cc:237] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.197638 8015 catalog_manager.cc:5495] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260620 02:29:17.225440 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.219961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.97.2:38733)
0620 02:29:17.220202 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.220207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.220225 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:17.220431 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.220435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.220532 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.220664 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:17.221061 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.221829 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.223275 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.223723 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.223827 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.224236 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:17.224841 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:17.224956 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.225131 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.225200 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:29:17.234731 8457 raft_consensus.cc:1261] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.234777 8173 raft_consensus.cc:1261] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.235806 8537 consensus_queue.cc:1035] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.236248 8533 consensus_queue.cc:1035] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.249712 8540 mvcc.cc:205] Tried to move back new op lower bound from 7298754794420928512 to 7298754794266988544. Current Snapshot: MvccSnapshot[applied={T|T < 7298754794420928512}]
W20260620 02:29:17.253578 8503 tablet.cc:2369] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:17.261262 8542 mvcc.cc:205] Tried to move back new op lower bound from 7298754794420928512 to 7298754794266988544. Current Snapshot: MvccSnapshot[applied={T|T < 7298754794420928512 or (T in {7298754794420928512})}]
I20260620 02:29:17.269914 8015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:17.270244 8015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:17.274932 8015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Sending DeleteTablet for 3 replicas of tablet bdba1e7a07024e86a9a124de3ad9c8c9
I20260620 02:29:17.276299 8292 tablet_service.cc:1514] Processing DeleteTablet for tablet bdba1e7a07024e86a9a124de3ad9c8c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51398
I20260620 02:29:17.276975 8432 tablet_service.cc:1514] Processing DeleteTablet for tablet bdba1e7a07024e86a9a124de3ad9c8c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52202
I20260620 02:29:17.277165 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet bdba1e7a07024e86a9a124de3ad9c8c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49546
I20260620 02:29:17.277403 8549 tablet_replica.cc:330] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: stopping tablet replica
I20260620 02:29:17.277987 8549 raft_consensus.cc:2229] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:17.278965 8549 raft_consensus.cc:2258] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:17.281909 8549 ts_tablet_manager.cc:1905] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:17.283308 8015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:17.284117 8015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:17.293766 8549 ts_tablet_manager.cc:1918] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:17.294214 8549 log.cc:1196] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/bdba1e7a07024e86a9a124de3ad9c8c9
I20260620 02:29:17.295054 8549 ts_tablet_manager.cc:1939] T bdba1e7a07024e86a9a124de3ad9c8c9 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting consensus metadata
I20260620 02:29:17.295938 8431 tablet_service.cc:1467] Processing CreateTablet for tablet 6286bc2c91984b259448db73078513c3 (DEFAULT_TABLE table=test-table [id=4f3c6accb19248829e437f0130beb444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.296301 8152 tablet_service.cc:1467] Processing CreateTablet for tablet 6286bc2c91984b259448db73078513c3 (DEFAULT_TABLE table=test-table [id=4f3c6accb19248829e437f0130beb444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.296541 8431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6286bc2c91984b259448db73078513c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.296847 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6286bc2c91984b259448db73078513c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.297470 8001 catalog_manager.cc:4841] TS 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733): tablet bdba1e7a07024e86a9a124de3ad9c8c9 (table test-table [id=2fab63cacd544b2caad37213e3da1a30]) successfully deleted
I20260620 02:29:17.301404 8292 tablet_service.cc:1467] Processing CreateTablet for tablet 6286bc2c91984b259448db73078513c3 (DEFAULT_TABLE table=test-table [id=4f3c6accb19248829e437f0130beb444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.301944 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6286bc2c91984b259448db73078513c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.305328 8528 tablet_bootstrap.cc:492] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Bootstrap starting.
I20260620 02:29:17.307183 8528 tablet_bootstrap.cc:654] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.312793 8526 tablet_bootstrap.cc:492] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Bootstrap starting.
I20260620 02:29:17.314612 8526 tablet_bootstrap.cc:654] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.316272 8527 tablet_bootstrap.cc:492] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Bootstrap starting.
I20260620 02:29:17.318171 8527 tablet_bootstrap.cc:654] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.318603 8528 tablet_bootstrap.cc:492] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: No bootstrap required, opened a new log
I20260620 02:29:17.318765 8528 ts_tablet_manager.cc:1397] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260620 02:29:17.319790 8528 raft_consensus.cc:348] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.320111 8528 raft_consensus.cc:374] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.320197 8528 raft_consensus.cc:729] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7bcd3f16d841ae96b3dfc05fe8d5ac, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.320645 8528 consensus_queue.cc:260] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.321657 8528 ts_tablet_manager.cc:1428] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:29:17.324859 8551 tablet_replica.cc:330] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: stopping tablet replica
I20260620 02:29:17.325516 8551 raft_consensus.cc:2229] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:17.326090 8551 raft_consensus.cc:2258] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:17.327039 8550 tablet_replica.cc:330] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: stopping tablet replica
I20260620 02:29:17.327657 8550 raft_consensus.cc:2229] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:17.328277 8550 raft_consensus.cc:2258] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:17.330796 8551 ts_tablet_manager.cc:1905] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:17.330796 8550 ts_tablet_manager.cc:1905] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:17.333475 8526 tablet_bootstrap.cc:492] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: No bootstrap required, opened a new log
I20260620 02:29:17.333740 8526 ts_tablet_manager.cc:1397] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20260620 02:29:17.335845 8526 raft_consensus.cc:348] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.336203 8526 raft_consensus.cc:374] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.336349 8526 raft_consensus.cc:729] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.336717 8526 consensus_queue.cc:260] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.338611 8551 ts_tablet_manager.cc:1918] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:17.339318 8551 log.cc:1196] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/bdba1e7a07024e86a9a124de3ad9c8c9
I20260620 02:29:17.340111 8551 ts_tablet_manager.cc:1939] T bdba1e7a07024e86a9a124de3ad9c8c9 P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting consensus metadata
I20260620 02:29:17.341050 8526 ts_tablet_manager.cc:1428] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 02:29:17.342308 8002 catalog_manager.cc:4841] TS 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453): tablet bdba1e7a07024e86a9a124de3ad9c8c9 (table test-table [id=2fab63cacd544b2caad37213e3da1a30]) successfully deleted
I20260620 02:29:17.342630 8550 ts_tablet_manager.cc:1918] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:17.342833 8550 log.cc:1196] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/bdba1e7a07024e86a9a124de3ad9c8c9
I20260620 02:29:17.343588 8550 ts_tablet_manager.cc:1939] T bdba1e7a07024e86a9a124de3ad9c8c9 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting consensus metadata
I20260620 02:29:17.345489 8527 tablet_bootstrap.cc:492] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: No bootstrap required, opened a new log
I20260620 02:29:17.345485 8003 catalog_manager.cc:4841] TS cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025): tablet bdba1e7a07024e86a9a124de3ad9c8c9 (table test-table [id=2fab63cacd544b2caad37213e3da1a30]) successfully deleted
I20260620 02:29:17.345640 8527 ts_tablet_manager.cc:1397] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.014s
I20260620 02:29:17.346436 8527 raft_consensus.cc:348] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.346683 8527 raft_consensus.cc:374] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.346772 8527 raft_consensus.cc:729] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac14a0cc8f5493a8ced732c66eb78da, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.347103 8527 consensus_queue.cc:260] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.347718 8527 ts_tablet_manager.cc:1428] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 02:29:17.401063 8224 tablet.cc:2369] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:17.424299 8537 raft_consensus.cc:482] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:17.424669 8537 raft_consensus.cc:504] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.425565 8537 leader_election.cc:288] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.426191 8456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6286bc2c91984b259448db73078513c3" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" is_pre_election: true
I20260620 02:29:17.426198 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6286bc2c91984b259448db73078513c3" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" is_pre_election: true
I20260620 02:29:17.426486 8173 raft_consensus.cc:2454] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:17.426506 8456 raft_consensus.cc:2454] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:17.426936 8245 leader_election.cc:302] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, 6ac14a0cc8f5493a8ced732c66eb78da; no voters:
I20260620 02:29:17.427289 8537 raft_consensus.cc:2781] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:17.427400 8537 raft_consensus.cc:482] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:17.427538 8537 raft_consensus.cc:3037] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.429836 8537 raft_consensus.cc:504] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.430440 8537 leader_election.cc:288] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Requested vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.430975 8456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6286bc2c91984b259448db73078513c3" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
I20260620 02:29:17.430958 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6286bc2c91984b259448db73078513c3" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
I20260620 02:29:17.431180 8456 raft_consensus.cc:3037] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.431180 8173 raft_consensus.cc:3037] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.434216 8173 raft_consensus.cc:2454] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:17.434300 8456 raft_consensus.cc:2454] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:17.434691 8246 leader_election.cc:302] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, cf7bcd3f16d841ae96b3dfc05fe8d5ac; no voters:
I20260620 02:29:17.435142 8537 raft_consensus.cc:2781] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:17.435367 8537 raft_consensus.cc:686] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Becoming Leader. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Running, Role: LEADER
I20260620 02:29:17.435693 8537 consensus_queue.cc:237] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.438732 8015 catalog_manager.cc:5495] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:17.504459 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.487878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.0.97.62:40939)
0620 02:29:17.488153 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.488162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.488801 (+ 639us) server_negotiation.cc:407] Connection header received
0620 02:29:17.489047 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.489053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.489135 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.489250 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:17.490564 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.491634 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.493033 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.493637 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.493784 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.497716 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:17.497722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:17.497725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:17.497760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:17.501218 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:17.502255 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:17.502259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:17.502262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:17.502412 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:17.502841 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:17.502846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:17.502849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:17.503200 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:17.503374 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.503802 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.504049 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 02:29:17.504580 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.487885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40939 (local address 127.0.0.1:48448)
0620 02:29:17.488657 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.488675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.488870 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.489335 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.489339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.490073 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:17.490393 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.490403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.491796 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.491800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.492839 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.492852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.493861 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.493867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.493997 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.494788 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:17.494806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:17.497453 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:17.501376 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:17.501393 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:17.501402 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:17.502062 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:17.502550 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:17.502555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:17.502558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:17.502707 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:17.503364 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:17.503372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:17.503544 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.503727 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.504182 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":153,"thread_start_us":446,"threads_started":1}
I20260620 02:29:17.510517 8015 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48448:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:29:17.511597 8015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:17.519033 8153 tablet_service.cc:1467] Processing CreateTablet for tablet 107006748e534e8db8c788dadbe64316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50f33b8dacf544f09f069c62ea85a6da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:29:17.519647 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107006748e534e8db8c788dadbe64316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.519465 8292 tablet_service.cc:1467] Processing CreateTablet for tablet 107006748e534e8db8c788dadbe64316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50f33b8dacf544f09f069c62ea85a6da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:29:17.519559 8432 tablet_service.cc:1467] Processing CreateTablet for tablet 107006748e534e8db8c788dadbe64316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50f33b8dacf544f09f069c62ea85a6da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:29:17.520066 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107006748e534e8db8c788dadbe64316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.520068 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107006748e534e8db8c788dadbe64316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.527354 8526 tablet_bootstrap.cc:492] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Bootstrap starting.
I20260620 02:29:17.527490 8527 tablet_bootstrap.cc:492] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: Bootstrap starting.
I20260620 02:29:17.528544 8528 tablet_bootstrap.cc:492] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Bootstrap starting.
I20260620 02:29:17.529211 8526 tablet_bootstrap.cc:654] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.529383 8527 tablet_bootstrap.cc:654] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.530329 8528 tablet_bootstrap.cc:654] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.540091 8527 tablet_bootstrap.cc:492] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: No bootstrap required, opened a new log
I20260620 02:29:17.540355 8527 ts_tablet_manager.cc:1397] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260620 02:29:17.540804 8528 tablet_bootstrap.cc:492] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: No bootstrap required, opened a new log
I20260620 02:29:17.541075 8528 ts_tablet_manager.cc:1397] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260620 02:29:17.541376 8527 raft_consensus.cc:348] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.541699 8526 tablet_bootstrap.cc:492] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: No bootstrap required, opened a new log
I20260620 02:29:17.541709 8527 raft_consensus.cc:374] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.541889 8527 raft_consensus.cc:729] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac14a0cc8f5493a8ced732c66eb78da, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.541911 8526 ts_tablet_manager.cc:1397] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260620 02:29:17.541940 8528 raft_consensus.cc:348] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.542198 8528 raft_consensus.cc:374] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.542281 8528 raft_consensus.cc:729] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7bcd3f16d841ae96b3dfc05fe8d5ac, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.542387 8527 consensus_queue.cc:260] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.542757 8526 raft_consensus.cc:348] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.543025 8526 raft_consensus.cc:374] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.543120 8526 raft_consensus.cc:729] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.542930 8534 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.543314 8534 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.543440 8526 consensus_queue.cc:260] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.542563 8528 consensus_queue.cc:260] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.543949 8537 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.544229 8537 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.544255 8528 ts_tablet_manager.cc:1428] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:29:17.544600 8527 ts_tablet_manager.cc:1428] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:29:17.545022 8532 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.545871 8532 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.546456 8526 ts_tablet_manager.cc:1428] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:29:17.564956 8537 raft_consensus.cc:482] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:17.565223 8537 raft_consensus.cc:504] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.565979 8537 leader_election.cc:288] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.566565 8456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107006748e534e8db8c788dadbe64316" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" is_pre_election: true
I20260620 02:29:17.566581 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107006748e534e8db8c788dadbe64316" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" is_pre_election: true
I20260620 02:29:17.566864 8456 raft_consensus.cc:2454] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:17.566871 8173 raft_consensus.cc:2454] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:17.567348 8245 leader_election.cc:302] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, 6ac14a0cc8f5493a8ced732c66eb78da; no voters:
I20260620 02:29:17.567705 8537 raft_consensus.cc:2781] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:17.567955 8537 raft_consensus.cc:482] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:17.568059 8537 raft_consensus.cc:3037] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.571077 8537 raft_consensus.cc:504] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.571739 8537 leader_election.cc:288] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Requested vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.572263 8456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107006748e534e8db8c788dadbe64316" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
I20260620 02:29:17.572268 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107006748e534e8db8c788dadbe64316" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
I20260620 02:29:17.572523 8456 raft_consensus.cc:3037] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.572525 8173 raft_consensus.cc:3037] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.574715 8456 raft_consensus.cc:2454] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:17.575371 8246 leader_election.cc:302] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, cf7bcd3f16d841ae96b3dfc05fe8d5ac; no voters:
I20260620 02:29:17.575709 8537 raft_consensus.cc:2781] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:17.575733 8173 raft_consensus.cc:2454] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:17.575906 8537 raft_consensus.cc:686] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Becoming Leader. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Running, Role: LEADER
I20260620 02:29:17.576191 8537 consensus_queue.cc:237] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.578030 8533 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: TxnStatusTablet state changed. Reason: New leader 0f96c24cfe4e4ae8ace9c7b795a7a519. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.578352 8533 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:29:17.579097 8566 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:29:17.579381 8566 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:29:17.580289 8016 catalog_manager.cc:5495] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260620 02:29:17.609687 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.603312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38733 (local address 127.0.0.1:51416)
0620 02:29:17.603564 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.603584 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.603723 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.604163 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.604166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.604176 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:29:17.604339 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.604346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.605987 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.605992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.607249 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.607262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.608357 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.608361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.608527 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.608599 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:29:17.609209 (+ 610us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:29:17.609231 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.609306 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.609436 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:29:17.609781 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.603311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.97.2:38733)
0620 02:29:17.603521 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.603528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.603656 (+ 128us) server_negotiation.cc:407] Connection header received
0620 02:29:17.603917 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.603923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.604016 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.604127 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:17.604509 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.605792 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.607391 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.608147 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.608294 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.608876 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:17.609074 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:17.609171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.609360 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.609448 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:29:17.614471 8173 raft_consensus.cc:1261] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.614704 8456 raft_consensus.cc:1261] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.615187 8537 consensus_queue.cc:1035] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.615548 8533 consensus_queue.cc:1035] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.622145 8534 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: TxnStatusTablet state changed. Reason: New leader 0f96c24cfe4e4ae8ace9c7b795a7a519. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.622507 8534 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.624109 8532 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: TxnStatusTablet state changed. Reason: New leader 0f96c24cfe4e4ae8ace9c7b795a7a519. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.624430 8532 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.626559 8537 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.626869 8537 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:29:17.628305 8537 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.628549 8537 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 0f96c24cfe4e4ae8ace9c7b795a7a519: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:29:17.628890 8541 mvcc.cc:205] Tried to move back new op lower bound from 7298754795982049280 to 7298754795834859520. Current Snapshot: MvccSnapshot[applied={T|T < 7298754795982049280}]
I20260620 02:29:17.629317 8534 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.629547 8534 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P 6ac14a0cc8f5493a8ced732c66eb78da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.631291 8532 tablet_replica.cc:439] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } } }
I20260620 02:29:17.631572 8532 tablet_replica.cc:442] T 107006748e534e8db8c788dadbe64316 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:29:17.653625 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.646329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40259 (local address 127.0.97.2:38733)
0620 02:29:17.646583 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.646590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.647357 (+ 767us) server_negotiation.cc:407] Connection header received
0620 02:29:17.647529 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.647535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.647627 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.647752 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:17.648336 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.649508 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.651107 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.651917 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.652083 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.652690 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:17.652848 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:17.652952 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.653159 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.653294 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":86}
I20260620 02:29:17.653683 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.646447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38733 (local address 127.0.97.2:40259)
0620 02:29:17.647238 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.647254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.647395 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.647783 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.647787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.647800 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:29:17.648140 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.648153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.649686 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.649690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.650947 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.650960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.652168 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.652174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.652389 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.652464 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:29:17.652974 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:29:17.652984 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.653114 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.653264 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":2,"thread_start_us":423,"threads_started":1}
I20260620 02:29:17.664848 8173 raft_consensus.cc:1261] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.664927 8456 raft_consensus.cc:1261] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.665514 8533 consensus_queue.cc:1035] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.665861 8537 consensus_queue.cc:1035] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.734130 8016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:17.734403 8016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:17.738126 8016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Sending DeleteTablet for 3 replicas of tablet 6286bc2c91984b259448db73078513c3
I20260620 02:29:17.739270 8432 tablet_service.cc:1514] Processing DeleteTablet for tablet 6286bc2c91984b259448db73078513c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52202
I20260620 02:29:17.739239 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet 6286bc2c91984b259448db73078513c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49546
I20260620 02:29:17.739581 8550 tablet_replica.cc:330] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: stopping tablet replica
I20260620 02:29:17.739591 8551 tablet_replica.cc:330] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: stopping tablet replica
I20260620 02:29:17.739807 8550 raft_consensus.cc:2229] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:17.739823 8551 raft_consensus.cc:2229] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:17.740028 8550 raft_consensus.cc:2258] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:17.740051 8551 raft_consensus.cc:2258] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:17.740361 8292 tablet_service.cc:1514] Processing DeleteTablet for tablet 6286bc2c91984b259448db73078513c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51398
I20260620 02:29:17.740688 8549 tablet_replica.cc:330] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: stopping tablet replica
I20260620 02:29:17.740895 8549 raft_consensus.cc:2229] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:17.741284 8549 raft_consensus.cc:2258] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:17.741534 8550 ts_tablet_manager.cc:1905] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:17.742503 8549 ts_tablet_manager.cc:1905] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:17.748692 8550 ts_tablet_manager.cc:1918] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:29:17.748880 8550 log.cc:1196] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/6286bc2c91984b259448db73078513c3
I20260620 02:29:17.749624 8550 ts_tablet_manager.cc:1939] T 6286bc2c91984b259448db73078513c3 P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting consensus metadata
I20260620 02:29:17.750046 8551 ts_tablet_manager.cc:1905] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:17.751005 8003 catalog_manager.cc:4841] TS cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025): tablet 6286bc2c91984b259448db73078513c3 (table test-table [id=4f3c6accb19248829e437f0130beb444]) successfully deleted
I20260620 02:29:17.754531 8549 ts_tablet_manager.cc:1918] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:29:17.754786 8549 log.cc:1196] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/6286bc2c91984b259448db73078513c3
I20260620 02:29:17.755281 8549 ts_tablet_manager.cc:1939] T 6286bc2c91984b259448db73078513c3 P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting consensus metadata
I20260620 02:29:17.756441 8551 ts_tablet_manager.cc:1918] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:29:17.756527 8001 catalog_manager.cc:4841] TS 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733): tablet 6286bc2c91984b259448db73078513c3 (table test-table [id=4f3c6accb19248829e437f0130beb444]) successfully deleted
I20260620 02:29:17.756611 8551 log.cc:1196] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/6286bc2c91984b259448db73078513c3
I20260620 02:29:17.757053 8551 ts_tablet_manager.cc:1939] T 6286bc2c91984b259448db73078513c3 P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting consensus metadata
I20260620 02:29:17.758059 8002 catalog_manager.cc:4841] TS 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453): tablet 6286bc2c91984b259448db73078513c3 (table test-table [id=4f3c6accb19248829e437f0130beb444]) successfully deleted
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:29:17.805966 8389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:17.794100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.0.97.62:40939)
0620 02:29:17.794328 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.794336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.794708 (+ 372us) server_negotiation.cc:407] Connection header received
0620 02:29:17.794883 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.794890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.794984 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.795096 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:17.796514 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.797738 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.799299 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.800012 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.800145 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.805666 (+ 5521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48464: BlockingRecv error: recv got EOF from 127.0.0.1:48464 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 02:29:17 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:29:17.824839 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.809027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.97.62:40939)
0620 02:29:17.809222 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.809228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.809668 (+ 440us) server_negotiation.cc:407] Connection header received
0620 02:29:17.809841 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.809845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.809912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.809993 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:17.811543 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.812353 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.814195 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.814616 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.814715 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.819335 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:17.819341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:17.819344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:17.819395 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:17.822075 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:17.822794 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:17.822799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:17.822801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:17.822946 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:17.823375 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:17.823379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:17.823381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:17.823681 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:17.823942 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.824271 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.824506 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:29:17.828775 8016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:17.829596 8016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:17.836745 8292 tablet_service.cc:1467] Processing CreateTablet for tablet d68cb515cd474e5aafba468bca7087ae (DEFAULT_TABLE table=test-table [id=918276f40f074d0ebabf0d2dbb145e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.836705 8153 tablet_service.cc:1467] Processing CreateTablet for tablet d68cb515cd474e5aafba468bca7087ae (DEFAULT_TABLE table=test-table [id=918276f40f074d0ebabf0d2dbb145e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.837221 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68cb515cd474e5aafba468bca7087ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.837227 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68cb515cd474e5aafba468bca7087ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.837987 8432 tablet_service.cc:1467] Processing CreateTablet for tablet d68cb515cd474e5aafba468bca7087ae (DEFAULT_TABLE table=test-table [id=918276f40f074d0ebabf0d2dbb145e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:17.838429 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68cb515cd474e5aafba468bca7087ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:17.844082 8527 tablet_bootstrap.cc:492] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Bootstrap starting.
I20260620 02:29:17.844185 8526 tablet_bootstrap.cc:492] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Bootstrap starting.
I20260620 02:29:17.845814 8526 tablet_bootstrap.cc:654] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.845868 8527 tablet_bootstrap.cc:654] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.849191 8528 tablet_bootstrap.cc:492] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Bootstrap starting.
I20260620 02:29:17.851399 8528 tablet_bootstrap.cc:654] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:17.856688 8527 tablet_bootstrap.cc:492] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: No bootstrap required, opened a new log
I20260620 02:29:17.856927 8527 ts_tablet_manager.cc:1397] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260620 02:29:17.857041 8528 tablet_bootstrap.cc:492] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: No bootstrap required, opened a new log
I20260620 02:29:17.857221 8528 ts_tablet_manager.cc:1397] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 02:29:17.857839 8527 raft_consensus.cc:348] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.858177 8527 raft_consensus.cc:374] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.858295 8527 raft_consensus.cc:729] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac14a0cc8f5493a8ced732c66eb78da, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.858562 8527 consensus_queue.cc:260] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.858728 8528 raft_consensus.cc:348] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.859011 8528 raft_consensus.cc:374] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.859112 8528 raft_consensus.cc:729] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7bcd3f16d841ae96b3dfc05fe8d5ac, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.859148 8527 ts_tablet_manager.cc:1428] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:29:17.859336 8528 consensus_queue.cc:260] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.860539 8528 ts_tablet_manager.cc:1428] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:29:17.861280 8526 tablet_bootstrap.cc:492] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: No bootstrap required, opened a new log
I20260620 02:29:17.861471 8526 ts_tablet_manager.cc:1397] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20260620 02:29:17.862361 8526 raft_consensus.cc:348] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.862645 8526 raft_consensus.cc:374] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:17.862813 8526 raft_consensus.cc:729] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Initialized, Role: FOLLOWER
I20260620 02:29:17.863095 8526 consensus_queue.cc:260] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.863742 8526 ts_tablet_manager.cc:1428] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:29:17.890996 8534 raft_consensus.cc:482] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:17.891243 8534 raft_consensus.cc:504] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.892776 8534 leader_election.cc:288] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.914775 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.893335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51157 (local address 127.0.97.3:39025)
0620 02:29:17.905544 (+ 12209us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.905552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.906470 (+ 918us) server_negotiation.cc:407] Connection header received
0620 02:29:17.906748 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.906754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.906831 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.906941 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.908088 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.909286 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.912076 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.913513 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.913617 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.913711 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.914258 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.914416 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12041,"thread_start_us":11856,"threads_started":1}
I20260620 02:29:17.915675 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.893207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39025 (local address 127.0.97.1:51157)
0620 02:29:17.906360 (+ 13153us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.906378 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.906588 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.907265 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.907276 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.907303 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.907899 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.907911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.909466 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.909471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.911877 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.911892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.913884 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.913890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.914083 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.914117 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.915063 (+ 946us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.915215 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13026,"mutex_wait_us":244,"thread_start_us":12592,"threads_started":1}
I20260620 02:29:17.916160 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.893917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46419 (local address 127.0.97.2:38733)
0620 02:29:17.894115 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.894122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.906066 (+ 11944us) server_negotiation.cc:407] Connection header received
0620 02:29:17.906470 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.906478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.906556 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.906621 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:17.907519 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.908799 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.911771 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.913305 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.913437 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.913547 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.915716 (+ 2169us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.915863 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:29:17.916486 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb515cd474e5aafba468bca7087ae" candidate_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" is_pre_election: true
I20260620 02:29:17.916757 8457 raft_consensus.cc:2454] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac14a0cc8f5493a8ced732c66eb78da in term 0.
I20260620 02:29:17.917063 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.893207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38733 (local address 127.0.97.1:46419)
0620 02:29:17.905839 (+ 12632us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:17.905868 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:17.906210 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:17.906810 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:17.906817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:17.906842 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:17.907224 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.907238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.909069 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.909075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.911568 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:17.911584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.915316 (+ 3732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.915322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:17.915509 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.915544 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:29:17.916419 (+ 875us) client_negotiation.cc:240] Negotiation successful
0620 02:29:17.916574 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12399,"mutex_wait_us":35,"spinlock_wait_cycles":8448,"thread_start_us":12365,"threads_started":1}
I20260620 02:29:17.917375 8106 leader_election.cc:302] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac14a0cc8f5493a8ced732c66eb78da, cf7bcd3f16d841ae96b3dfc05fe8d5ac; no voters:
I20260620 02:29:17.917807 8534 raft_consensus.cc:2781] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:17.917959 8534 raft_consensus.cc:482] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:17.918076 8534 raft_consensus.cc:3037] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.917958 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb515cd474e5aafba468bca7087ae" candidate_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" is_pre_election: true
I20260620 02:29:17.918444 8317 raft_consensus.cc:2454] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac14a0cc8f5493a8ced732c66eb78da in term 0.
I20260620 02:29:17.920979 8534 raft_consensus.cc:504] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.922184 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb515cd474e5aafba468bca7087ae" candidate_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519"
I20260620 02:29:17.922286 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb515cd474e5aafba468bca7087ae" candidate_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
I20260620 02:29:17.922487 8457 raft_consensus.cc:3037] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.922423 8317 raft_consensus.cc:3037] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:17.925482 8457 raft_consensus.cc:2454] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac14a0cc8f5493a8ced732c66eb78da in term 1.
I20260620 02:29:17.925925 8106 leader_election.cc:302] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac14a0cc8f5493a8ced732c66eb78da, cf7bcd3f16d841ae96b3dfc05fe8d5ac; no voters:
I20260620 02:29:17.925969 8317 raft_consensus.cc:2454] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac14a0cc8f5493a8ced732c66eb78da in term 1.
I20260620 02:29:17.926759 8534 leader_election.cc:288] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [CANDIDATE]: Term 1 election: Requested vote from peers 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:17.926865 8597 raft_consensus.cc:2781] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:17.927104 8597 raft_consensus.cc:686] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 LEADER]: Becoming Leader. State: Replica: 6ac14a0cc8f5493a8ced732c66eb78da, State: Running, Role: LEADER
I20260620 02:29:17.927597 8597 consensus_queue.cc:237] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:17.931478 8015 catalog_manager.cc:5495] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260620 02:29:17.963346 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:17.956177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.0.97.1:44453)
0620 02:29:17.956975 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:17.956981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:17.957040 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:29:17.957160 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:17.957165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:17.957276 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:17.957474 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:17.957926 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.959131 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.960499 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:17.961221 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:17.961351 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:17.961820 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:17.962627 (+ 807us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:17.962767 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:17.962854 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 02:29:17.962941 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":481,"threads_started":1}
I20260620 02:29:17.967700 8457 raft_consensus.cc:1261] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Refusing update from remote peer 6ac14a0cc8f5493a8ced732c66eb78da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.968047 8317 raft_consensus.cc:1261] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Refusing update from remote peer 6ac14a0cc8f5493a8ced732c66eb78da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:17.968528 8597 consensus_queue.cc:1035] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.968966 8534 consensus_queue.cc:1035] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:17.994882 8013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:17.995180 8013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:17.998739 8013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Sending DeleteTablet for 3 replicas of tablet d68cb515cd474e5aafba468bca7087ae
I20260620 02:29:17.999574 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet d68cb515cd474e5aafba468bca7087ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49546
I20260620 02:29:17.999967 8292 tablet_service.cc:1514] Processing DeleteTablet for tablet d68cb515cd474e5aafba468bca7087ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51398
I20260620 02:29:18.000322 8549 tablet_replica.cc:330] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: stopping tablet replica
I20260620 02:29:18.000509 8549 raft_consensus.cc:2229] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:18.000804 8549 raft_consensus.cc:2258] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:18.001173 8551 tablet_replica.cc:330] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: stopping tablet replica
I20260620 02:29:18.001360 8551 raft_consensus.cc:2229] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:18.001910 8551 raft_consensus.cc:2258] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:18.002059 8432 tablet_service.cc:1514] Processing DeleteTablet for tablet d68cb515cd474e5aafba468bca7087ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52202
I20260620 02:29:18.002349 8550 tablet_replica.cc:330] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: stopping tablet replica
I20260620 02:29:18.002539 8550 raft_consensus.cc:2229] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:18.002777 8550 raft_consensus.cc:2258] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:18.004417 8013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:18.005195 8013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:18.008462 8549 ts_tablet_manager.cc:1905] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:18.013500 8292 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba (DEFAULT_TABLE table=test-table [id=2b0137902e4a4b4dbe631217c064dd86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:18.013671 8432 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba (DEFAULT_TABLE table=test-table [id=2b0137902e4a4b4dbe631217c064dd86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:18.014138 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:18.014372 8292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:18.017515 8550 ts_tablet_manager.cc:1905] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:18.018175 8153 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba (DEFAULT_TABLE table=test-table [id=2b0137902e4a4b4dbe631217c064dd86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:18.018612 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:18.024729 8550 ts_tablet_manager.cc:1918] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:18.024945 8550 log.cc:1196] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/d68cb515cd474e5aafba468bca7087ae
I20260620 02:29:18.025497 8550 ts_tablet_manager.cc:1939] T d68cb515cd474e5aafba468bca7087ae P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting consensus metadata
I20260620 02:29:18.025803 8551 ts_tablet_manager.cc:1905] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:18.029475 8003 catalog_manager.cc:4841] TS cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025): tablet d68cb515cd474e5aafba468bca7087ae (table test-table [id=918276f40f074d0ebabf0d2dbb145e21]) successfully deleted
I20260620 02:29:18.034305 8549 ts_tablet_manager.cc:1918] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:18.034489 8549 log.cc:1196] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/d68cb515cd474e5aafba468bca7087ae
I20260620 02:29:18.035009 8549 ts_tablet_manager.cc:1939] T d68cb515cd474e5aafba468bca7087ae P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting consensus metadata
I20260620 02:29:18.035638 8526 tablet_bootstrap.cc:492] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Bootstrap starting.
I20260620 02:29:18.038400 8526 tablet_bootstrap.cc:654] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:18.038789 8528 tablet_bootstrap.cc:492] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Bootstrap starting.
I20260620 02:29:18.040511 8528 tablet_bootstrap.cc:654] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:18.043664 8001 catalog_manager.cc:4841] TS 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733): tablet d68cb515cd474e5aafba468bca7087ae (table test-table [id=918276f40f074d0ebabf0d2dbb145e21]) successfully deleted
I20260620 02:29:18.044445 8527 tablet_bootstrap.cc:492] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Bootstrap starting.
I20260620 02:29:18.047134 8527 tablet_bootstrap.cc:654] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:18.051766 8526 tablet_bootstrap.cc:492] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: No bootstrap required, opened a new log
I20260620 02:29:18.052038 8526 ts_tablet_manager.cc:1397] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260620 02:29:18.054607 8526 raft_consensus.cc:348] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.054917 8526 raft_consensus.cc:374] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:18.055018 8526 raft_consensus.cc:729] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Initialized, Role: FOLLOWER
I20260620 02:29:18.055327 8526 consensus_queue.cc:260] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.057492 8551 ts_tablet_manager.cc:1918] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:18.057719 8551 log.cc:1196] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/d68cb515cd474e5aafba468bca7087ae
I20260620 02:29:18.058825 8526 ts_tablet_manager.cc:1428] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260620 02:29:18.058497 8527 tablet_bootstrap.cc:492] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: No bootstrap required, opened a new log
I20260620 02:29:18.059249 8527 ts_tablet_manager.cc:1397] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260620 02:29:18.060195 8527 raft_consensus.cc:348] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.060525 8527 raft_consensus.cc:374] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:18.060667 8527 raft_consensus.cc:729] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac14a0cc8f5493a8ced732c66eb78da, State: Initialized, Role: FOLLOWER
I20260620 02:29:18.061038 8527 consensus_queue.cc:260] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.063382 8527 ts_tablet_manager.cc:1428] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 02:29:18.063602 8551 ts_tablet_manager.cc:1939] T d68cb515cd474e5aafba468bca7087ae P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting consensus metadata
I20260620 02:29:18.064901 8002 catalog_manager.cc:4841] TS 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453): tablet d68cb515cd474e5aafba468bca7087ae (table test-table [id=918276f40f074d0ebabf0d2dbb145e21]) successfully deleted
I20260620 02:29:18.067651 8528 tablet_bootstrap.cc:492] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: No bootstrap required, opened a new log
I20260620 02:29:18.067839 8528 ts_tablet_manager.cc:1397] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20260620 02:29:18.068681 8528 raft_consensus.cc:348] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.068949 8528 raft_consensus.cc:374] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:18.069079 8528 raft_consensus.cc:729] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7bcd3f16d841ae96b3dfc05fe8d5ac, State: Initialized, Role: FOLLOWER
I20260620 02:29:18.069350 8528 consensus_queue.cc:260] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.069958 8528 ts_tablet_manager.cc:1428] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:29:18.091440 8578 raft_consensus.cc:482] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:18.091720 8578 raft_consensus.cc:504] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.092623 8578 leader_election.cc:288] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:18.093318 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad19d76fd3f4b6e8335b529cb1dd2ba" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" is_pre_election: true
I20260620 02:29:18.093357 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad19d76fd3f4b6e8335b529cb1dd2ba" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" is_pre_election: true
I20260620 02:29:18.093572 8173 raft_consensus.cc:2454] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:18.093624 8457 raft_consensus.cc:2454] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 0.
I20260620 02:29:18.094157 8246 leader_election.cc:302] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, cf7bcd3f16d841ae96b3dfc05fe8d5ac; no voters:
I20260620 02:29:18.094520 8578 raft_consensus.cc:2781] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:18.094647 8578 raft_consensus.cc:482] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:18.094743 8578 raft_consensus.cc:3037] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:18.098165 8578 raft_consensus.cc:504] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.099577 8457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad19d76fd3f4b6e8335b529cb1dd2ba" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac"
I20260620 02:29:18.099682 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad19d76fd3f4b6e8335b529cb1dd2ba" candidate_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac14a0cc8f5493a8ced732c66eb78da"
I20260620 02:29:18.099819 8457 raft_consensus.cc:3037] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:18.099920 8173 raft_consensus.cc:3037] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:18.103534 8173 raft_consensus.cc:2454] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:18.104029 8245 leader_election.cc:302] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f96c24cfe4e4ae8ace9c7b795a7a519, 6ac14a0cc8f5493a8ced732c66eb78da; no voters:
I20260620 02:29:18.104355 8537 raft_consensus.cc:2781] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:18.104506 8537 raft_consensus.cc:686] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Becoming Leader. State: Replica: 0f96c24cfe4e4ae8ace9c7b795a7a519, State: Running, Role: LEADER
I20260620 02:29:18.104839 8457 raft_consensus.cc:2454] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f96c24cfe4e4ae8ace9c7b795a7a519 in term 1.
I20260620 02:29:18.104764 8537 consensus_queue.cc:237] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } }
I20260620 02:29:18.105633 8578 leader_election.cc:288] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [CANDIDATE]: Term 1 election: Requested vote from peers 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453), cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025)
I20260620 02:29:18.108489 8013 catalog_manager.cc:5495] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f96c24cfe4e4ae8ace9c7b795a7a519" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260620 02:29:18.146842 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:18.139909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.0.97.2:38733)
0620 02:29:18.140086 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:18.140093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:18.140273 (+ 180us) server_negotiation.cc:407] Connection header received
0620 02:29:18.140469 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:18.140475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:18.140563 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:18.140671 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:18.141091 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.142271 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.144177 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.144913 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.145113 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:18.145643 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:18.145821 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:18.145931 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:18.146414 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 02:29:18.146500 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:29:18.158742 8172 raft_consensus.cc:1261] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:18.159528 8573 consensus_queue.cc:1035] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac14a0cc8f5493a8ced732c66eb78da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:18.160732 8457 raft_consensus.cc:1261] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Refusing update from remote peer 0f96c24cfe4e4ae8ace9c7b795a7a519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:18.161342 8537 consensus_queue.cc:1035] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7bcd3f16d841ae96b3dfc05fe8d5ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:18.244105 8013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:18.244450 8013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:18.247910 8013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f5a5a2c7c54422a885e130a5a4e9c7: Sending DeleteTablet for 3 replicas of tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba
I20260620 02:29:18.248798 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49546
I20260620 02:29:18.249226 8551 tablet_replica.cc:330] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: stopping tablet replica
I20260620 02:29:18.249197 8432 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52202
I20260620 02:29:18.249447 8551 raft_consensus.cc:2229] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:18.249573 8550 tablet_replica.cc:330] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: stopping tablet replica
I20260620 02:29:18.249650 8551 raft_consensus.cc:2258] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:18.249791 8550 raft_consensus.cc:2229] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:18.249989 8550 raft_consensus.cc:2258] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:18.250998 8551 ts_tablet_manager.cc:1905] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:18.251226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 8087
I20260620 02:29:18.257256 8291 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51398
I20260620 02:29:18.257586 8549 tablet_replica.cc:330] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: stopping tablet replica
I20260620 02:29:18.257818 8549 raft_consensus.cc:2229] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:18.258176 8549 raft_consensus.cc:2258] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:18.264544 8550 ts_tablet_manager.cc:1905] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:18.265642 8549 ts_tablet_manager.cc:1905] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:18.271817 8550 ts_tablet_manager.cc:1918] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:29:18.272004 8550 log.cc:1196] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/2ad19d76fd3f4b6e8335b529cb1dd2ba
I20260620 02:29:18.272430 8550 ts_tablet_manager.cc:1939] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P cf7bcd3f16d841ae96b3dfc05fe8d5ac: Deleting consensus metadata
I20260620 02:29:18.273679 8549 ts_tablet_manager.cc:1918] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:29:18.274379 8551 ts_tablet_manager.cc:1918] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:29:18.274550 8551 log.cc:1196] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/2ad19d76fd3f4b6e8335b529cb1dd2ba
I20260620 02:29:18.274685 8549 log.cc:1196] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/2ad19d76fd3f4b6e8335b529cb1dd2ba
I20260620 02:29:18.275049 8551 ts_tablet_manager.cc:1939] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 6ac14a0cc8f5493a8ced732c66eb78da: Deleting consensus metadata
I20260620 02:29:18.275260 8549 ts_tablet_manager.cc:1939] T 2ad19d76fd3f4b6e8335b529cb1dd2ba P 0f96c24cfe4e4ae8ace9c7b795a7a519: Deleting consensus metadata
I20260620 02:29:18.276288 8002 catalog_manager.cc:4841] TS 6ac14a0cc8f5493a8ced732c66eb78da (127.0.97.1:44453): tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba (table test-table [id=2b0137902e4a4b4dbe631217c064dd86]) successfully deleted
I20260620 02:29:18.277689 8001 catalog_manager.cc:4841] TS 0f96c24cfe4e4ae8ace9c7b795a7a519 (127.0.97.2:38733): tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba (table test-table [id=2b0137902e4a4b4dbe631217c064dd86]) successfully deleted
I20260620 02:29:18.279234 8003 catalog_manager.cc:4841] TS cf7bcd3f16d841ae96b3dfc05fe8d5ac (127.0.97.3:39025): tablet 2ad19d76fd3f4b6e8335b529cb1dd2ba (table test-table [id=2b0137902e4a4b4dbe631217c064dd86]) successfully deleted
Jun 20 02:29:18 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 02:29:18.302006 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:18.257035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.0.97.1:44453)
0620 02:29:18.257227 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:18.257237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:18.281513 (+ 24276us) server_negotiation.cc:407] Connection header received
0620 02:29:18.281952 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:18.281958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:18.282037 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:18.282112 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:18.283792 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.284884 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.286742 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.287357 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.287480 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:18.293231 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:18.293243 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:18.293260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:18.293326 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:18.297811 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:18.298890 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:18.298900 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:18.298905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:18.299085 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:18.299542 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:18.299553 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:18.299556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:18.300211 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:18.300650 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:18.301097 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 02:29:18.301341 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:29:18.303037 8218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:18.428913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 8087
I20260620 02:29:18.450551 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 8227
Jun 20 02:29:18 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 02:29:18.468226 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:18.451437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.0.97.2:38733)
0620 02:29:18.451635 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:18.451641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:18.451663 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:29:18.451819 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:18.451824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:18.451911 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:18.452015 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:18.453398 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.454525 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.455871 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.456588 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.456704 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:18.461075 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:18.461095 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:18.461116 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:18.461159 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:18.465287 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:18.466021 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:18.466031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:18.466035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:18.466208 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:18.466556 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:18.466562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:18.466565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:18.467064 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:18.467357 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:18.467639 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 02:29:18.467862 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:29:18.469056 8357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:18.615890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 8227
I20260620 02:29:18.641852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 8366
Jun 20 02:29:18 dist-test-slave-r706 krb5kdc[7971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:29:18.657925 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:18.642677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.97.3:39025)
0620 02:29:18.643199 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:18.643204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:18.643221 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:29:18.643280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:18.643283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:18.643344 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:18.643441 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:18.644701 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.645476 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.646709 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:18.647105 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:18.647202 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:18.651937 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:18.651955 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:18.651972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:18.652022 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:18.655333 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:18.656041 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:18.656048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:18.656051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:18.656234 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:18.656565 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:18.656568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:18.656570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:18.656923 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:18.657254 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:18.657455 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 02:29:18.657647 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":307,"threads_started":1}
I20260620 02:29:18.658582 8497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:18.758626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 8366
I20260620 02:29:18.776343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 7984
I20260620 02:29:18.777627 8056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:18.896162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 7984
2026-06-20T02:29:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5490 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:29:19 dist-test-slave-r706 krb5kdc[8619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:29:19 dist-test-slave-r706 krb5kdc[8619](info): set up 2 sockets
Jun 20 02:29:19 dist-test-slave-r706 krb5kdc[8619](info): commencing operation
krb5kdc: starting...
W20260620 02:29:20.034051 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:29:20 dist-test-slave-r706 krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:29:20.286911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38569
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:20.620540 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:20.620853 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:20.621060 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:20.628552 8632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:20.628691 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:20.628782 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:20.628870 8632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:20.628971 8632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:20.639910 8632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:20.641379 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:20.642752 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:20.655105 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:20.655151 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:20.655743 8638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:20.656009 8632 server_base.cc:1034] running on GCE node
I20260620 02:29:20.656893 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:20.658433 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:20.659663 8632 hybrid_clock.cc:648] HybridClock initialized: now 1781922560659612 us; error 38 us; skew 500 ppm
I20260620 02:29:20.660013 8632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T02:29:20Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1828 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:29:20 dist-test-slave-r706 krb5kdc[8646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:29:20 dist-test-slave-r706 krb5kdc[8646](info): set up 2 sockets
Jun 20 02:29:20 dist-test-slave-r706 krb5kdc[8646](info): commencing operation
krb5kdc: starting...
W20260620 02:29:21.862375 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:29:22 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:29:22.125541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:22.477780 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:22.478111 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:22.478266 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:22.485720 8659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:22.485843 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:22.485962 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:22.486068 8659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:22.486156 8659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:22.496380 8659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:22.497884 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:22.499285 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:22.512041 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:22.512112 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:22.512697 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:22.513527 8659 server_base.cc:1034] running on GCE node
I20260620 02:29:22.514395 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:22.515928 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:22.517150 8659 hybrid_clock.cc:648] HybridClock initialized: now 1781922562517107 us; error 49 us; skew 500 ppm
I20260620 02:29:22.517544 8659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:22 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:22.520921 8659 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:29:22.522730 8659 webserver.cc:460] Webserver started at http://127.0.97.62:35305/ using document root <none> and password file <none>
I20260620 02:29:22.523504 8659 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:22.523674 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:22.524032 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:22.527419 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "0048f09941aa45709738170ec1aa0555"
format_stamp: "Formatted at 2026-06-20 02:29:22 on dist-test-slave-r706"
server_key: "8caec445e06b1365e652f1a2afe04c17"
server_key_iv: "83b23cd33cda91c2888507af098588ff"
server_key_version: "encryptionkey@0"
I20260620 02:29:22.528307 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "0048f09941aa45709738170ec1aa0555"
format_stamp: "Formatted at 2026-06-20 02:29:22 on dist-test-slave-r706"
server_key: "8caec445e06b1365e652f1a2afe04c17"
server_key_iv: "83b23cd33cda91c2888507af098588ff"
server_key_version: "encryptionkey@0"
I20260620 02:29:22.534169 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 02:29:22.538379 8674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:22.539917 8659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:29:22.540159 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "0048f09941aa45709738170ec1aa0555"
format_stamp: "Formatted at 2026-06-20 02:29:22 on dist-test-slave-r706"
server_key: "8caec445e06b1365e652f1a2afe04c17"
server_key_iv: "83b23cd33cda91c2888507af098588ff"
server_key_version: "encryptionkey@0"
I20260620 02:29:22.540400 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:22.558351 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:22.562717 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:22.563050 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:22.584168 8659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39433
I20260620 02:29:22.584225 8725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39433 every 8 connection(s)
I20260620 02:29:22.585860 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:22.589963 8726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:22.591148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 8659
I20260620 02:29:22.591355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:29:22.591995 388 external_mini_cluster.cc:1428] Setting key a684ee6fca41394fcc78db8885ca663d
I20260620 02:29:22.605795 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Bootstrap starting.
Jun 20 02:29:22 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:22.609966 8726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:22.611178 8726 log.cc:824] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:22.614948 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: No bootstrap required, opened a new log
I20260620 02:29:22.619174 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:22.594164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.0.97.62:39433)
0620 02:29:22.595107 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:22.595120 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:22.595167 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:29:22.596964 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:22.597061 (+ 97us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:22.597609 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:22.598006 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:22.600176 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:22.601747 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:22.603754 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:22.604558 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:22.604743 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:22.609855 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:22.609868 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:22.609885 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:22.609934 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:22.613894 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:22.614768 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:22.614778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:22.614830 (+ 52us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:22.615093 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:22.615514 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:22.615519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:22.615522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:22.616287 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:22.616593 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:22.617137 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 02:29:22.617654 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":450,"threads_started":1}
I20260620 02:29:22.624987 8726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:22.625526 8726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:22.625662 8726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0048f09941aa45709738170ec1aa0555, State: Initialized, Role: FOLLOWER
I20260620 02:29:22.626359 8726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:22.626662 8726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:22.626816 8726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:22.626993 8726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:22.629362 8726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:22.629931 8726 leader_election.cc:302] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0048f09941aa45709738170ec1aa0555; no voters:
I20260620 02:29:22.630584 8726 leader_election.cc:288] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:29:22.630741 8731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:22.631676 8731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 LEADER]: Becoming Leader. State: Replica: 0048f09941aa45709738170ec1aa0555, State: Running, Role: LEADER
I20260620 02:29:22.632396 8731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:22.632997 8726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:22.636024 8733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0048f09941aa45709738170ec1aa0555. Latest consensus state: current_term: 1 leader_uuid: "0048f09941aa45709738170ec1aa0555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } } }
I20260620 02:29:22.636255 8732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0048f09941aa45709738170ec1aa0555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } } }
I20260620 02:29:22.636617 8732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:22.636623 8733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:22.638639 8739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:22.646543 8739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:22.657953 8739 catalog_manager.cc:1347] Generated new cluster ID: 0800dd7ea513476a983ba595f9d0d7ee
I20260620 02:29:22.658195 8739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:22.696138 8739 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:29:22.698156 8739 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:22.719496 8739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Generated new TSK 0
I20260620 02:29:22.720495 8739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:29:22.798970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39433
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:23.155812 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:23.156179 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:23.156363 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:23.164094 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:23.164281 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:23.164469 8752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:23.174916 8752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:23.176322 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:23.177675 8752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:23.191629 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:23.191708 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:23.191708 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:23.192772 8752 server_base.cc:1034] running on GCE node
I20260620 02:29:23.193589 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:23.194654 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:23.195864 8752 hybrid_clock.cc:648] HybridClock initialized: now 1781922563195821 us; error 45 us; skew 500 ppm
I20260620 02:29:23.196225 8752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:23 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:23.199822 8752 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:29:23.201736 8752 webserver.cc:460] Webserver started at http://127.0.97.1:36859/ using document root <none> and password file <none>
I20260620 02:29:23.202523 8752 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:23.202677 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:23.203053 8752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:23.206459 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "fece3cf801694b6cbb2e7cd87d5e88c4"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "4249c8c6c5efbec63248c871b936ceb9"
server_key_iv: "49b0ff028887557b7e3d9557bb8e57e4"
server_key_version: "encryptionkey@0"
I20260620 02:29:23.207417 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fece3cf801694b6cbb2e7cd87d5e88c4"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "4249c8c6c5efbec63248c871b936ceb9"
server_key_iv: "49b0ff028887557b7e3d9557bb8e57e4"
server_key_version: "encryptionkey@0"
I20260620 02:29:23.213694 8752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:29:23.217684 8767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:23.219199 8752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 02:29:23.219437 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "fece3cf801694b6cbb2e7cd87d5e88c4"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "4249c8c6c5efbec63248c871b936ceb9"
server_key_iv: "49b0ff028887557b7e3d9557bb8e57e4"
server_key_version: "encryptionkey@0"
I20260620 02:29:23.219678 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:23.234086 8752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:23.238588 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:23.238916 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:23.239982 8752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:23.241429 8752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:23.241546 8752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:23.241703 8752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:23.241782 8752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:23.279647 8752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40347
I20260620 02:29:23.279695 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40347 every 8 connection(s)
I20260620 02:29:23.280972 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:23.289189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 8752
I20260620 02:29:23.289454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:29:23.290272 388 external_mini_cluster.cc:1428] Setting key 6863e2ecefc594ec1862e25b931ce493
Jun 20 02:29:23 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:23.304610 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:23.283708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46245 (local address 127.0.97.62:39433)
0620 02:29:23.284246 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:23.284251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:23.285264 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 02:29:23.287362 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:23.287369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:23.287479 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:23.287621 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:23.289900 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.290754 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.292978 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.293439 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.293544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:23.298498 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:23.298504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:23.298508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:23.298546 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:23.301291 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:23.302129 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:23.302134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:23.302137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:23.302261 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:23.302733 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:23.302742 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:23.302744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:23.303032 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:23.303205 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:23.303953 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 02:29:23.304209 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":298,"threads_started":1}
I20260620 02:29:23.305694 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:23.284132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:46245)
0620 02:29:23.285066 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:23.285116 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:23.287024 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:23.287907 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:23.287918 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:23.288581 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:23.289624 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:23.289659 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.290975 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.290982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:23.292813 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:23.292833 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.293714 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.293726 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:23.294102 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:23.294995 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:23.295036 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:23.298210 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:23.301469 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:23.301479 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:23.301500 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:23.301974 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:23.302381 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:23.302386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:23.302390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:23.302605 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:23.303169 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:23.303179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:23.303560 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 02:29:23.303837 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 02:29:23.304406 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":361,"threads_started":1}
I20260620 02:29:23.307489 8880 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39433
I20260620 02:29:23.307816 8880 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:23.308578 8880 heartbeater.cc:511] Master 127.0.97.62:39433 requested a full tablet report, sending...
I20260620 02:29:23.311194 8691 ts_manager.cc:194] Registered new tserver with Master: fece3cf801694b6cbb2e7cd87d5e88c4 (127.0.97.1:40347)
I20260620 02:29:23.313617 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46245
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:29:23.349179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39433
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:23.690104 8886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:23.690510 8886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:23.690711 8886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:23.698343 8886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:23.698558 8886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:23.698787 8886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:23.710261 8886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:23.711721 8886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:23.713200 8886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:23.727445 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:23.727373 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:23.727401 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:23.728077 8886 server_base.cc:1034] running on GCE node
I20260620 02:29:23.728703 8886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:23.729748 8886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:23.730953 8886 hybrid_clock.cc:648] HybridClock initialized: now 1781922563730908 us; error 38 us; skew 500 ppm
I20260620 02:29:23.731269 8886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:23 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:23.734653 8886 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:29:23.736176 8886 webserver.cc:460] Webserver started at http://127.0.97.2:43707/ using document root <none> and password file <none>
I20260620 02:29:23.736861 8886 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:23.736975 8886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:23.737366 8886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:23.740677 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "3a1b2d59593745d18dc36704bc283e9a"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "254ca4facf06da0f76420ea0107c3481"
server_key_iv: "4fc858729acc052068636b15f60ced46"
server_key_version: "encryptionkey@0"
I20260620 02:29:23.741602 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a1b2d59593745d18dc36704bc283e9a"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "254ca4facf06da0f76420ea0107c3481"
server_key_iv: "4fc858729acc052068636b15f60ced46"
server_key_version: "encryptionkey@0"
I20260620 02:29:23.747602 8886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:29:23.751317 8901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:23.752689 8886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:29:23.752872 8886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "3a1b2d59593745d18dc36704bc283e9a"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "254ca4facf06da0f76420ea0107c3481"
server_key_iv: "4fc858729acc052068636b15f60ced46"
server_key_version: "encryptionkey@0"
I20260620 02:29:23.753154 8886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:23.767784 8886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:23.772022 8886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:23.772356 8886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:23.773437 8886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:23.774840 8886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:23.774926 8886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:23.775118 8886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:23.775183 8886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:23.810873 8886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38265
I20260620 02:29:23.810907 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38265 every 8 connection(s)
I20260620 02:29:23.812374 8886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:23.814224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 8886
I20260620 02:29:23.814447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:29:23.814934 388 external_mini_cluster.cc:1428] Setting key 0f668ed0e52cf0255c68248a3a561eab
Jun 20 02:29:23 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:23.834239 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:23.815190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54773 (local address 127.0.97.62:39433)
0620 02:29:23.815756 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:23.815762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:23.816733 (+ 971us) server_negotiation.cc:407] Connection header received
0620 02:29:23.818853 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:23.818858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:23.818947 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:23.819077 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:23.820647 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.821450 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.823059 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.823509 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.823669 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:23.827924 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:23.827929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:23.827932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:23.827967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:23.830712 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:23.831548 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:23.831556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:23.831564 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:23.831700 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:23.832163 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:23.832170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:23.832173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:23.832449 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:23.832623 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:23.833515 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 02:29:23.833768 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":321,"threads_started":1}
I20260620 02:29:23.835291 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:23.815712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:54773)
0620 02:29:23.816570 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:23.816624 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:23.818573 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:23.819333 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:23.819342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:23.819870 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:23.820466 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:23.820485 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.821627 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.821631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:23.822913 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:23.822925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:23.823766 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:23.823774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:23.824032 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:23.824873 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:23.824909 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:23.827662 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:23.830885 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:23.830895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:23.830916 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:23.831405 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:23.831823 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:23.831829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:23.831832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:23.832045 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:23.832595 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:23.832604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:23.833122 (+ 518us) client_negotiation.cc:769] Sending connection context
0620 02:29:23.833421 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:29:23.834027 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":354,"threads_started":1}
I20260620 02:29:23.837332 9014 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39433
I20260620 02:29:23.837677 9014 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:29:23.838441 9014 heartbeater.cc:511] Master 127.0.97.62:39433 requested a full tablet report, sending...
I20260620 02:29:23.840159 8691 ts_manager.cc:194] Registered new tserver with Master: 3a1b2d59593745d18dc36704bc283e9a (127.0.97.2:38265)
I20260620 02:29:23.841120 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:29:23.871654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39433
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:24.219265 9020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:24.219655 9020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:24.219835 9020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:24.227602 9020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:24.227782 9020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:24.227978 9020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:24.238435 9020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:24.239904 9020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:24.241284 9020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:24.256204 9025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:24.256263 9028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:24.256114 9026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:24.257405 9020 server_base.cc:1034] running on GCE node
I20260620 02:29:24.258358 9020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:24.260150 9020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:24.261410 9020 hybrid_clock.cc:648] HybridClock initialized: now 1781922564261304 us; error 103 us; skew 500 ppm
I20260620 02:29:24.261795 9020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:24 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:24.265308 9020 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:29:24.267023 9020 webserver.cc:460] Webserver started at http://127.0.97.3:33623/ using document root <none> and password file <none>
I20260620 02:29:24.267766 9020 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:24.267926 9020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:24.268263 9020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:24.271492 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "0ba4fea07c1841278fc5544318dc26e4"
format_stamp: "Formatted at 2026-06-20 02:29:24 on dist-test-slave-r706"
server_key: "06dc49e0ef853e31c8913bd9cb41da54"
server_key_iv: "5dc00be86def6a33d70340441f70c940"
server_key_version: "encryptionkey@0"
I20260620 02:29:24.272372 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0ba4fea07c1841278fc5544318dc26e4"
format_stamp: "Formatted at 2026-06-20 02:29:24 on dist-test-slave-r706"
server_key: "06dc49e0ef853e31c8913bd9cb41da54"
server_key_iv: "5dc00be86def6a33d70340441f70c940"
server_key_version: "encryptionkey@0"
I20260620 02:29:24.277907 9020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:29:24.282724 9035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:24.284487 9020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:29:24.284687 9020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "0ba4fea07c1841278fc5544318dc26e4"
format_stamp: "Formatted at 2026-06-20 02:29:24 on dist-test-slave-r706"
server_key: "06dc49e0ef853e31c8913bd9cb41da54"
server_key_iv: "5dc00be86def6a33d70340441f70c940"
server_key_version: "encryptionkey@0"
I20260620 02:29:24.284943 9020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:24.308410 9020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:24.313328 9020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:24.313606 9020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:24.314620 9020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:24.316027 9020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:24.316121 9020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:24.316272 9020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:24.316336 9020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:24.317210 8880 heartbeater.cc:503] Master 127.0.97.62:39433 was elected leader, sending a full tablet report...
I20260620 02:29:24.353467 9020 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37745
I20260620 02:29:24.353569 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37745 every 8 connection(s)
I20260620 02:29:24.354875 9020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:24.358422 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9020
I20260620 02:29:24.358645 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:29:24.359194 388 external_mini_cluster.cc:1428] Setting key 2cf663cac5af141be2bb11f3e16bf07e
Jun 20 02:29:24 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:24.380195 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:24.358455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46621 (local address 127.0.97.62:39433)
0620 02:29:24.361241 (+ 2786us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:24.361248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:24.361272 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:29:24.363137 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:24.363141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:24.363237 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:24.363341 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:24.365374 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.366425 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.368168 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.368750 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.368904 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:24.373150 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:24.373157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:24.373161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:24.373214 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:24.376672 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.377777 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.377783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.377786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.377951 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.378453 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.378457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.378459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.378763 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:24.378901 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:24.379690 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 02:29:24.379912 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2654,"thread_start_us":2482,"threads_started":1}
I20260620 02:29:24.381312 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:24.359170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:46621)
0620 02:29:24.360419 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:24.360491 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:24.362798 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:24.363717 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:24.363727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:24.364337 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:24.365081 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:24.365124 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.366646 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.366650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:24.367992 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:24.368008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.369205 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.369212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:24.369465 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:24.370208 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:24.370232 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:24.372685 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:24.376868 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:24.376878 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:24.376898 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:24.377565 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:24.378076 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:24.378080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:24.378082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:24.378295 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:24.378895 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:24.378902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:24.379276 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 02:29:24.379530 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 02:29:24.379967 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":457,"threads_started":1}
I20260620 02:29:24.382987 9148 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39433
I20260620 02:29:24.383345 9148 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:24.384166 9148 heartbeater.cc:511] Master 127.0.97.62:39433 requested a full tablet report, sending...
I20260620 02:29:24.386133 8691 ts_manager.cc:194] Registered new tserver with Master: 0ba4fea07c1841278fc5544318dc26e4 (127.0.97.3:37745)
I20260620 02:29:24.387288 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46621
I20260620 02:29:24.389103 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:29:24.569594 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 8752
Jun 20 02:29:24 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:29:24.590441 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:24.571173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.0.97.1:40347)
0620 02:29:24.572117 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:24.572124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:24.572152 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:24.572214 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:24.572221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:24.572443 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:24.572590 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:24.574241 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.575315 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.576689 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.577632 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.577752 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:24.582678 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:24.582697 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:24.582714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:24.582764 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:24.586988 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.587853 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.587862 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.587864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.588055 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.588521 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.588527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.588531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.589069 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:24.589391 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:24.589712 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:29:24.590010 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":597,"threads_started":1}
I20260620 02:29:24.591852 8875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:24.677676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 8752
I20260620 02:29:24.695928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 8886
Jun 20 02:29:24 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:29:24.713881 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:24.696980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.2:38265)
0620 02:29:24.697564 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:24.697572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:24.697600 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:24.697664 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:24.697671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:24.697890 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:24.698023 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:24.699425 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.700439 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.701931 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.702550 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.702661 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:24.707305 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:24.707330 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:24.707347 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:24.707390 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:24.710573 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.711557 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.711567 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.711569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.711727 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.712131 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.712135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.712137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.712526 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:24.712778 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:24.713214 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 02:29:24.713455 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":235,"threads_started":1}
I20260620 02:29:24.715270 9009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:24.802816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 8886
I20260620 02:29:24.820379 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9020
Jun 20 02:29:24 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:29:24.838968 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:24.821593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.0.97.3:37745)
0620 02:29:24.822108 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:24.822115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:24.822146 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:29:24.822207 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:24.822214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:24.822457 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:24.822589 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:24.823955 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.825086 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.826392 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:24.826970 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:24.827079 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:24.831935 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:24.831952 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:24.831968 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:24.832017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:24.835284 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.836162 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.836171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.836173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.836329 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:24.836895 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:24.836899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:24.836905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:24.837483 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:24.837769 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:24.838099 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:29:24.838340 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260620 02:29:24.840550 9143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:24.928316 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9020
I20260620 02:29:24.948776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 8659
I20260620 02:29:24.951066 8721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:25.069077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 8659
I20260620 02:29:25.091511 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39433
--webserver_interface=127.0.97.62
--webserver_port=35305
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:25.448149 9166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:25.448482 9166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:25.448577 9166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:25.456054 9166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:25.456197 9166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:25.456249 9166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:25.456373 9166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:25.456449 9166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:25.467088 9166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:25.468524 9166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:25.469975 9166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:25.482510 9174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:25.482738 9172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:25.483144 9171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:25.483445 9166 server_base.cc:1034] running on GCE node
I20260620 02:29:25.484287 9166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:25.485894 9166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:25.487108 9166 hybrid_clock.cc:648] HybridClock initialized: now 1781922565487060 us; error 35 us; skew 500 ppm
I20260620 02:29:25.487491 9166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:25 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:25.490808 9166 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:29:25.492483 9166 webserver.cc:460] Webserver started at http://127.0.97.62:35305/ using document root <none> and password file <none>
I20260620 02:29:25.493324 9166 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:25.493498 9166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:25.499348 9166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:29:25.502514 9181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:25.504103 9166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:29:25.504300 9166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "0048f09941aa45709738170ec1aa0555"
format_stamp: "Formatted at 2026-06-20 02:29:22 on dist-test-slave-r706"
server_key: "8caec445e06b1365e652f1a2afe04c17"
server_key_iv: "83b23cd33cda91c2888507af098588ff"
server_key_version: "encryptionkey@0"
I20260620 02:29:25.505434 9166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:25.540855 9166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:25.545411 9166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:25.545754 9166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:25.565738 9166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39433
I20260620 02:29:25.565810 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39433 every 8 connection(s)
I20260620 02:29:25.567416 9166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:25.569113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9166
I20260620 02:29:25.570299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40347
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39433
--webserver_port=36859
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:29:25.578830 9233 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:29:25.582773 9233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Bootstrap starting.
I20260620 02:29:25.602846 9233 log.cc:824] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:25.612675 9233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:29:25.613371 9233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Bootstrap complete.
I20260620 02:29:25.622954 9233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:25.623920 9233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0048f09941aa45709738170ec1aa0555, State: Initialized, Role: FOLLOWER
I20260620 02:29:25.624588 9233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:25.624845 9233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:25.624984 9233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:25.625222 9233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:29:25.629354 9233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:25.629973 9233 leader_election.cc:302] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0048f09941aa45709738170ec1aa0555; no voters:
I20260620 02:29:25.630671 9233 leader_election.cc:288] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:29:25.630820 9237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:29:25.632109 9237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [term 2 LEADER]: Becoming Leader. State: Replica: 0048f09941aa45709738170ec1aa0555, State: Running, Role: LEADER
I20260620 02:29:25.632795 9237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } }
I20260620 02:29:25.633203 9233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:25.635560 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0048f09941aa45709738170ec1aa0555. Latest consensus state: current_term: 2 leader_uuid: "0048f09941aa45709738170ec1aa0555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } } }
I20260620 02:29:25.635967 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:25.635777 9238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0048f09941aa45709738170ec1aa0555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0048f09941aa45709738170ec1aa0555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39433 } } }
I20260620 02:29:25.636209 9238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:25.646103 9244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:25.653517 9244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:25.655419 9244 catalog_manager.cc:1259] Loaded cluster ID: 0800dd7ea513476a983ba595f9d0d7ee
I20260620 02:29:25.655646 9244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:25.661288 9244 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:25.663527 9244 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0048f09941aa45709738170ec1aa0555: Loaded TSK: 0
I20260620 02:29:25.665673 9244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:29:25.952616 9235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:25.953097 9235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:25.953339 9235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:25.966320 9235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:25.966485 9235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:25.966696 9235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:25.979796 9235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36859
--tserver_master_addrs=127.0.97.62:39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:25.981986 9235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:25.983863 9235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:25.999846 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:26.002367 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:26.002413 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:26.002743 9235 server_base.cc:1034] running on GCE node
I20260620 02:29:26.003724 9235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:26.005113 9235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:26.006405 9235 hybrid_clock.cc:648] HybridClock initialized: now 1781922566006335 us; error 65 us; skew 500 ppm
I20260620 02:29:26.006820 9235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:26.011278 9235 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:29:26.013204 9235 webserver.cc:460] Webserver started at http://127.0.97.1:36859/ using document root <none> and password file <none>
I20260620 02:29:26.014110 9235 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:26.014273 9235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:26.021840 9235 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 02:29:26.025475 9269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:26.027311 9235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:29:26.027518 9235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "fece3cf801694b6cbb2e7cd87d5e88c4"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "4249c8c6c5efbec63248c871b936ceb9"
server_key_iv: "49b0ff028887557b7e3d9557bb8e57e4"
server_key_version: "encryptionkey@0"
I20260620 02:29:26.028970 9235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:26.054574 9235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:26.060782 9235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:26.061151 9235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:26.062672 9235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:26.064527 9235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:26.064647 9235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:26.064744 9235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:26.064816 9235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:26.100937 9235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40347
I20260620 02:29:26.101079 9381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40347 every 8 connection(s)
I20260620 02:29:26.102499 9235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:26.111433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9235
I20260620 02:29:26.112076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38265
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39433
--webserver_port=43707
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:26.121932 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:26.105958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:50881)
0620 02:29:26.106829 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:26.106866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:26.108795 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:26.110308 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:26.110319 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:26.110976 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:26.111559 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.111586 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.113403 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.113409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.114622 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.114646 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.115715 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.115722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.116010 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.116745 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:26.116770 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:26.120862 (+ 4092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":402,"threads_started":1}
W20260620 02:29:26.122553 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:26.105791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50881 (local address 127.0.97.62:39433)
0620 02:29:26.106586 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:26.106602 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:26.107049 (+ 447us) server_negotiation.cc:407] Connection header received
0620 02:29:26.109416 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:26.109429 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:26.109795 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:26.110189 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:26.111805 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.113231 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.114797 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.115449 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.115663 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.121393 (+ 5730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50881: BlockingRecv error: recv got EOF from 127.0.97.1:50881 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":253,"threads_started":1}
W20260620 02:29:26.122872 9382 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39433 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39433: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:26.132115 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:26.123572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:55863)
0620 02:29:26.123743 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:26.123763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:26.123868 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:26.124230 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:26.124233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:26.124550 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:26.124696 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.124703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.126237 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.126241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.127151 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.127163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.128195 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.128199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.128331 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.128965 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:26.128983 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:26.131909 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:29:26.132543 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:26.123679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55863 (local address 127.0.97.62:39433)
0620 02:29:26.123867 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:26.123875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:26.123899 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:29:26.123968 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:26.123976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:26.124071 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:26.124204 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:26.124840 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.126090 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.127300 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.128031 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.128197 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.132240 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55863: BlockingRecv error: recv got EOF from 127.0.97.1:55863 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:26.141752 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:26.133056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:55915)
0620 02:29:26.133215 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:26.133239 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:26.133324 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:26.133723 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:26.133729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:26.134003 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:26.134223 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.134232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.135935 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.135939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.136839 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.136850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.137918 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.137923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.138048 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.138654 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:26.138671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:26.141539 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 02:29:26.142055 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:26.133154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55915 (local address 127.0.97.62:39433)
0620 02:29:26.133341 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:26.133349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:26.133393 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:29:26.133457 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:26.133465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:26.133572 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:26.133688 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:26.134424 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.135787 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.136984 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.137775 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.137936 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.141792 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55915: BlockingRecv error: recv got EOF from 127.0.97.1:55915 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 02:29:26.142215 9382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:29:26.472950 9386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:26.473338 9386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:26.473439 9386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:26.482101 9386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:26.482326 9386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:26.482791 9386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:26.495157 9386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43707
--tserver_master_addrs=127.0.97.62:39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:26.496681 9386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:26.498579 9386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:26.513533 9391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:26.513482 9392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:26.513561 9394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:26.514549 9386 server_base.cc:1034] running on GCE node
I20260620 02:29:26.515308 9386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:26.516362 9386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:26.517647 9386 hybrid_clock.cc:648] HybridClock initialized: now 1781922566517591 us; error 47 us; skew 500 ppm
I20260620 02:29:26.518132 9386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:26.526113 9386 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:29:26.527801 9386 webserver.cc:460] Webserver started at http://127.0.97.2:43707/ using document root <none> and password file <none>
I20260620 02:29:26.528533 9386 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:26.528736 9386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:26.534638 9386 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 02:29:26.537760 9401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:26.539283 9386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:29:26.539489 9386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "3a1b2d59593745d18dc36704bc283e9a"
format_stamp: "Formatted at 2026-06-20 02:29:23 on dist-test-slave-r706"
server_key: "254ca4facf06da0f76420ea0107c3481"
server_key_iv: "4fc858729acc052068636b15f60ced46"
server_key_version: "encryptionkey@0"
I20260620 02:29:26.540510 9386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:26.562987 9386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:26.567610 9386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:26.567922 9386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:26.568981 9386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:26.570580 9386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:26.570712 9386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:26.570858 9386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:26.570972 9386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:26.605747 9386 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38265
I20260620 02:29:26.605849 9513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38265 every 8 connection(s)
I20260620 02:29:26.607234 9386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:26.610809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9386
I20260620 02:29:26.611507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37745
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39433
--webserver_port=33623
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:29:26.625931 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:26.610016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53201 (local address 127.0.97.62:39433)
0620 02:29:26.610206 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:26.610213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:26.611642 (+ 1429us) server_negotiation.cc:407] Connection header received
0620 02:29:26.613582 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:26.613591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:26.613699 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:26.613855 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:26.615400 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.616624 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.618408 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.619281 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.619401 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.625668 (+ 6267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53201: BlockingRecv error: recv got EOF from 127.0.97.2:53201 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:29:26.626698 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:26.610597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:53201)
0620 02:29:26.611444 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:26.611489 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:26.613288 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:26.614087 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:26.614096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:26.614626 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:26.615192 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.615210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.616843 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.616847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.618237 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.618251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.619564 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.619574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.619930 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.620764 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:26.620805 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:26.625461 (+ 4656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":361,"threads_started":1}
W20260620 02:29:26.627424 9514 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39433 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39433: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:26.637929 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:26.628119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:36019)
0620 02:29:26.628431 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:26.628447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:26.628547 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:26.628932 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:26.628935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:26.629339 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:26.629490 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.629498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.631114 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.631118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.632027 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.632039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.633317 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.633323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.633474 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.634243 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:26.634263 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:26.637677 (+ 3414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 02:29:26.638494 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:26.628214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36019 (local address 127.0.97.62:39433)
0620 02:29:26.628369 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:26.628373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:26.628531 (+ 158us) server_negotiation.cc:407] Connection header received
0620 02:29:26.628657 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:26.628665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:26.628757 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:26.628870 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:26.629647 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.630930 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.632203 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.633119 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.633279 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.637994 (+ 4715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36019: BlockingRecv error: recv got EOF from 127.0.97.2:36019 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:26.648051 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:26.638772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:38149)
0620 02:29:26.638962 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:26.638982 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:26.639093 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:26.639547 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:26.639552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:26.639934 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:26.640129 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.640140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.641373 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.641379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.642843 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:26.642857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.643575 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.643581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:26.643763 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.644422 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:26.644440 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:26.647699 (+ 3259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 02:29:26.648435 9514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:29:26.648569 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:26.639120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38149 (local address 127.0.97.62:39433)
0620 02:29:26.639278 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:26.639282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:26.639299 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:29:26.639348 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:26.639351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:26.639411 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:26.639501 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:26.640286 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.641235 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.643001 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:26.643435 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:26.643549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:26.647922 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38149: BlockingRecv error: recv got EOF from 127.0.97.2:38149 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 02:29:26.950937 9517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:26.951346 9517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:26.951530 9517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:26.962620 9517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:26.962805 9517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:26.962989 9517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:26.973676 9517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33623
--tserver_master_addrs=127.0.97.62:39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:26.975134 9517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:26.976503 9517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:26.990769 9523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:26.990914 9522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:26.990752 9525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:26.991652 9517 server_base.cc:1034] running on GCE node
I20260620 02:29:26.992321 9517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:26.993412 9517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:26.994644 9517 hybrid_clock.cc:648] HybridClock initialized: now 1781922566994608 us; error 43 us; skew 500 ppm
I20260620 02:29:26.994997 9517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:26 dist-test-slave-r706 krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:26.998327 9517 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:29:26.999847 9517 webserver.cc:460] Webserver started at http://127.0.97.3:33623/ using document root <none> and password file <none>
I20260620 02:29:27.000538 9517 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:27.000661 9517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:27.006561 9517 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:29:27.010363 9532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:27.012135 9517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:29:27.012379 9517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "0ba4fea07c1841278fc5544318dc26e4"
format_stamp: "Formatted at 2026-06-20 02:29:24 on dist-test-slave-r706"
server_key: "06dc49e0ef853e31c8913bd9cb41da54"
server_key_iv: "5dc00be86def6a33d70340441f70c940"
server_key_version: "encryptionkey@0"
I20260620 02:29:27.013545 9517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:27.036583 9517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:27.041447 9517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:27.041786 9517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:27.042958 9517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:27.044596 9517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:27.044688 9517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:27.044804 9517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:27.044904 9517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:27.080715 9517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37745
I20260620 02:29:27.080878 9644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37745 every 8 connection(s)
I20260620 02:29:27.082593 9517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:27.089938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9517
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:29:27.103390 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:27.085701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56849 (local address 127.0.97.62:39433)
0620 02:29:27.085942 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:27.085947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:27.087372 (+ 1425us) server_negotiation.cc:407] Connection header received
0620 02:29:27.089460 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:27.089466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:27.089587 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:27.089730 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:27.092054 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.093240 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.095816 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.096489 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.096640 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.103001 (+ 6361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56849: BlockingRecv error: recv got EOF from 127.0.97.3:56849 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 02:29:27.104099 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:27.086231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:56849)
0620 02:29:27.087134 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:27.087182 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:27.089117 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:27.090020 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:27.090030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:27.090826 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:27.091824 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.091856 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.093423 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.093428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.095637 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.095653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.096809 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.096815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.097133 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.097966 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:27.098009 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:27.102614 (+ 4605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":363,"threads_started":1}
W20260620 02:29:27.105129 9645 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39433 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39433: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260620 02:29:27.110817 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:27.091360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.0.97.62:39433)
0620 02:29:27.091952 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:27.091959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:27.091987 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:27.092051 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:27.092060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:27.092127 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:27.092215 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:27.094246 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.095395 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.097420 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.097978 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.098143 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.101441 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:27.101455 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:27.101482 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:27.101545 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:27.106322 (+ 4777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:27.107681 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:27.107695 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:27.107706 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:27.107954 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:27.108477 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:27.108482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:27.108484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:27.109188 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:27.109435 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:27.110056 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 02:29:27.110507 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":309,"threads_started":1}
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:27.115662 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:27.106121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:50363)
0620 02:29:27.106394 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:27.106415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:27.106529 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:27.106860 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:27.106865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:27.107250 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:27.107406 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.107416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.108976 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.108982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.110463 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.110478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.111388 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.111395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.111581 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.112211 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:27.112229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:27.115445 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260620 02:29:27.115958 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:27.106369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50363 (local address 127.0.97.62:39433)
0620 02:29:27.106576 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:27.106579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:27.106594 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:29:27.106646 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:27.106649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:27.106706 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:27.106825 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:27.107632 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.108745 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.110619 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.111225 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.111374 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.115675 (+ 4301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50363: BlockingRecv error: recv got EOF from 127.0.97.3:50363 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:27.125340 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:27.116485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:38319)
0620 02:29:27.116681 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:27.116721 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:27.116840 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:27.117232 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:27.117237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:27.117634 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:27.117883 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.117900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.119309 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.119313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.120268 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.120279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.120964 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.120969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.121168 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.121782 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:27.121800 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:27.124927 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:29:27.125780 9645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:29:27.125878 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:27.116596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38319 (local address 127.0.97.62:39433)
0620 02:29:27.116854 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:27.116859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:27.116879 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:29:27.116945 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:27.116950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:27.117092 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:27.117207 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:27.118076 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.119163 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.120413 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.120827 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.120926 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.125371 (+ 4445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38319: BlockingRecv error: recv got EOF from 127.0.97.3:38319 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:27.152489 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:27.142937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:45279)
0620 02:29:27.143486 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:27.143501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:27.143669 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:27.144033 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:27.144037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:27.144612 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:27.144829 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.144839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.146065 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.146069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.147165 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.147180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.147971 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.147977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.148153 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.148838 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:27.148857 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:27.152127 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":341,"threads_started":1}
W20260620 02:29:27.152613 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:27.143013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45279 (local address 127.0.97.62:39433)
0620 02:29:27.143291 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:27.143299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:27.143592 (+ 293us) server_negotiation.cc:407] Connection header received
0620 02:29:27.143824 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:27.143827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:27.143892 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:27.144000 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:27.144980 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.145879 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.147359 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.147775 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.147921 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.152371 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45279: BlockingRecv error: recv got EOF from 127.0.97.1:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:27 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:27.659965 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:27.649172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:51071)
0620 02:29:27.649814 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:27.649832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:27.650002 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:27.650624 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:27.650629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:27.651182 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:27.651502 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.651512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.652614 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.652618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.653862 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:27.653878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.654783 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.654789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:27.654966 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.655855 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:27.655878 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:27.659724 (+ 3846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":386,"threads_started":1}
W20260620 02:29:27.660351 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:27.649240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51071 (local address 127.0.97.62:39433)
0620 02:29:27.649415 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:27.649421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:27.649952 (+ 531us) server_negotiation.cc:407] Connection header received
0620 02:29:27.650292 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:27.650300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:27.650387 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:27.650556 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:27.651692 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.652422 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.654078 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:27.654533 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:27.654645 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:27.660079 (+ 5434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51071: BlockingRecv error: recv got EOF from 127.0.97.2:51071 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 02:29:28 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:28 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:28.135838 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:28.126421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:45175)
0620 02:29:28.127019 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:28.127043 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:28.127160 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:28.127497 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:28.127502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:28.127973 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:28.128187 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:28.128196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.129269 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.129274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:28.130378 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:28.130392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.131230 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.131236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:28.131375 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:28.132043 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:28.132063 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:28.135514 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":346,"threads_started":1}
W20260620 02:29:28.136053 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:28.126477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45175 (local address 127.0.97.62:39433)
0620 02:29:28.126665 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:28.126669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:28.127120 (+ 451us) server_negotiation.cc:407] Connection header received
0620 02:29:28.127260 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:28.127263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:28.127325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:28.127408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:28.128339 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.129104 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.130534 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.131012 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.131117 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:28.135783 (+ 4666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45175: BlockingRecv error: recv got EOF from 127.0.97.3:45175 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 02:29:28 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:28 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:28.162984 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:28.153832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:47453)
0620 02:29:28.154449 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:28.154466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:28.154607 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:28.154996 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:28.155000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:28.155425 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:28.155586 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:28.155595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.156728 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.156732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:28.157772 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:28.157786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.158525 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.158530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:28.158648 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:28.159350 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:28.159370 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:28.162736 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":368,"threads_started":1}
W20260620 02:29:28.163205 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:28.153834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47453 (local address 127.0.97.62:39433)
0620 02:29:28.154052 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:28.154057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:28.154564 (+ 507us) server_negotiation.cc:407] Connection header received
0620 02:29:28.154781 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:28.154785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:28.154851 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:28.154944 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:28.155752 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.156575 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.157912 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.158368 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.158474 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:28.162937 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47453: BlockingRecv error: recv got EOF from 127.0.97.1:47453 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 02:29:28 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:28 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:28.670296 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:28.661166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:57923)
0620 02:29:28.661836 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:28.661854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:28.661990 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:28.662314 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:28.662317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:28.662804 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:28.662990 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:28.663003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.664429 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.664433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:28.665657 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:28.665670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.666458 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.666463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:28.666603 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:28.667185 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:28.667201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:28.670056 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":424,"threads_started":1}
W20260620 02:29:28.670617 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:28.661274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57923 (local address 127.0.97.62:39433)
0620 02:29:28.661456 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:28.661461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:28.661943 (+ 482us) server_negotiation.cc:407] Connection header received
0620 02:29:28.662101 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:28.662105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:28.662172 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:28.662270 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:28.663156 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.664270 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.665793 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:28.666301 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:28.666453 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:28.670325 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57923: BlockingRecv error: recv got EOF from 127.0.97.2:57923 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 02:29:29 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:29 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:29.145828 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:29.136934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:35849)
0620 02:29:29.137529 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:29.137546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:29.137726 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:29.138105 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:29.138109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:29.138546 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:29.138692 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:29.138701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.139740 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.139747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:29.140654 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:29.140664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.141486 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.141492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:29.141662 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:29.142299 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:29.142318 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:29.145473 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":332,"threads_started":1}
W20260620 02:29:29.146016 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:29.137138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35849 (local address 127.0.97.62:39433)
0620 02:29:29.137390 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:29.137396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:29.137637 (+ 241us) server_negotiation.cc:407] Connection header received
0620 02:29:29.137860 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:29.137865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:29.137940 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:29.138048 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:29.138844 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.139588 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.140801 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.141323 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.141456 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:29.145741 (+ 4285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35849: BlockingRecv error: recv got EOF from 127.0.97.3:35849 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 02:29:29 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:29 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:29.173375 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:29.164238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:52449)
0620 02:29:29.164987 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:29.165042 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:29.165191 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:29.165619 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:29.165623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:29.166119 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:29.166289 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:29.166298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.167280 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.167283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:29.168168 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:29.168178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.168907 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.168911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:29.169127 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:29.169774 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:29.169793 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:29.172949 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":502,"threads_started":1}
W20260620 02:29:29.173564 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:29.164312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52449 (local address 127.0.97.62:39433)
0620 02:29:29.164606 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:29.164616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:29.165162 (+ 546us) server_negotiation.cc:407] Connection header received
0620 02:29:29.165337 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:29.165345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:29.165445 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:29.165576 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:29.166447 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.167160 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.168322 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.168763 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.168868 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:29.173295 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52449: BlockingRecv error: recv got EOF from 127.0.97.1:52449 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 02:29:29 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:29 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:29.679785 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:29.671259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:53829)
0620 02:29:29.671719 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:29.671736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:29.671858 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:29.672193 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:29.672197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:29.672576 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:29.672742 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:29.672750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.673835 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.673840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:29.674886 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:29.674899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.675628 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.675635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:29.675813 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:29.676488 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:29.676507 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:29.679506 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
W20260620 02:29:29.680083 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:29.671365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53829 (local address 127.0.97.62:39433)
0620 02:29:29.671624 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:29.671630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:29.671817 (+ 187us) server_negotiation.cc:407] Connection header received
0620 02:29:29.671962 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:29.671965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:29.672027 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:29.672109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:29.672879 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.673683 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.675040 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:29.675469 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:29.675643 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:29.679781 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53829: BlockingRecv error: recv got EOF from 127.0.97.2:53829 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 02:29:30 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:30 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:30.156410 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:30.147044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:42123)
0620 02:29:30.147756 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:30.147798 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:30.147992 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:30.148429 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:30.148434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:30.148855 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:30.149144 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:30.149156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.150194 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.150202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:30.151121 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:30.151133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.151850 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.151856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:30.152039 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:30.152718 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:30.152736 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:30.155993 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":417,"threads_started":1}
W20260620 02:29:30.156513 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:30.147168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42123 (local address 127.0.97.62:39433)
0620 02:29:30.147406 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:30.147415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:30.147932 (+ 517us) server_negotiation.cc:407] Connection header received
0620 02:29:30.148164 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:30.148168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:30.148247 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:30.148390 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:30.149300 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.150051 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.151275 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.151694 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.151834 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:30.156330 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42123: BlockingRecv error: recv got EOF from 127.0.97.3:42123 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 02:29:30 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:30 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:30.183092 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:30.174337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:57691)
0620 02:29:30.174868 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:30.174885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:30.175054 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:30.175438 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:30.175443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:30.175939 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:30.176095 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:30.176104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.177216 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.177222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:30.178185 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:30.178196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.178946 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.178951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:30.179155 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:30.179751 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:30.179771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:30.182839 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":284,"threads_started":1}
W20260620 02:29:30.183346 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:30.174419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57691 (local address 127.0.97.62:39433)
0620 02:29:30.174717 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:30.174724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:30.174978 (+ 254us) server_negotiation.cc:407] Connection header received
0620 02:29:30.175180 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:30.175185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:30.175255 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:30.175360 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:30.176255 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.177068 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.178342 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.178782 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.178918 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:30.183089 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57691: BlockingRecv error: recv got EOF from 127.0.97.1:57691 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 20 02:29:30 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:30 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:30.690004 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:30.680936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:36541)
0620 02:29:30.681546 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:30.681563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:30.681685 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:30.682028 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:30.682032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:30.682398 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:30.682580 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:30.682588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.683670 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.683677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:30.684822 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:30.684835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.685797 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.685805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:30.686042 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:30.686651 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:30.686677 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:30.689754 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":300,"threads_started":1}
W20260620 02:29:30.690325 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:30.681084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36541 (local address 127.0.97.62:39433)
0620 02:29:30.681260 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:30.681265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:30.681666 (+ 401us) server_negotiation.cc:407] Connection header received
0620 02:29:30.681787 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:30.681791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:30.681878 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:30.682009 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:30.682724 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.683517 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.684978 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:30.685630 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:30.685768 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:30.690016 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36541: BlockingRecv error: recv got EOF from 127.0.97.2:36541 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 02:29:31 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:31 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:31.166671 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:31.157668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:48691)
0620 02:29:31.158183 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:31.158200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:31.158322 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:31.158652 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:31.158655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:31.159039 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:31.159211 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:31.159220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.160575 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.160584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:31.161882 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:31.161895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.162622 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.162627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:31.162753 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:31.163381 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:31.163399 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:31.166437 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":294,"threads_started":1}
W20260620 02:29:31.166953 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:31.157796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48691 (local address 127.0.97.62:39433)
0620 02:29:31.158012 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:31.158017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:31.158289 (+ 272us) server_negotiation.cc:407] Connection header received
0620 02:29:31.158430 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:31.158433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:31.158502 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:31.158614 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:31.159369 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.160382 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.162047 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.162492 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.162600 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:31.166688 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48691: BlockingRecv error: recv got EOF from 127.0.97.3:48691 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 02:29:31 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:31 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:31.193132 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:31.184213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:33891)
0620 02:29:31.184767 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:31.184784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:31.184912 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:31.185331 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:31.185334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:31.185846 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:31.186018 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:31.186028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.187273 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.187277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:31.188201 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:31.188213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.188873 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.188877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:31.189037 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:31.189618 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:31.189637 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:31.192797 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":284,"threads_started":1}
W20260620 02:29:31.193459 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:31.184335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33891 (local address 127.0.97.62:39433)
0620 02:29:31.184563 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:31.184567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:31.184880 (+ 313us) server_negotiation.cc:407] Connection header received
0620 02:29:31.185073 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:31.185082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:31.185162 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:31.185247 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:31.186206 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.187136 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.188328 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.188741 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.188833 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:31.193080 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33891: BlockingRecv error: recv got EOF from 127.0.97.1:33891 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 02:29:31 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:31 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:31.703904 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:31.691043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:59679)
0620 02:29:31.691721 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:31.691739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:31.691913 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:31.692285 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:31.692290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:31.692708 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:31.692873 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:31.692881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.693964 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.693973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:31.694918 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:31.694930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.695716 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.695722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:31.695894 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:31.700616 (+ 4722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:31.700638 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:31.703654 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":313,"threads_started":1}
W20260620 02:29:31.704259 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:31.691325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59679 (local address 127.0.97.62:39433)
0620 02:29:31.691597 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:31.691603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:31.691831 (+ 228us) server_negotiation.cc:407] Connection header received
0620 02:29:31.692065 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:31.692069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:31.692142 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:31.692254 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:31.693049 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.693818 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.695068 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:31.695546 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:31.695695 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:31.703887 (+ 8192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59679: BlockingRecv error: recv got EOF from 127.0.97.2:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 20 02:29:32 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:32 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:32.176620 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:32.167738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:60431)
0620 02:29:32.168316 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:32.168334 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:32.168458 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:32.168769 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:32.168772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:32.169243 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:32.169426 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:32.169434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.170474 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.170478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:32.171416 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:32.171428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.172095 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.172099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:32.172228 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:32.172970 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:32.173020 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:32.176266 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":296,"threads_started":1}
W20260620 02:29:32.176731 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:32.167880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60431 (local address 127.0.97.62:39433)
0620 02:29:32.168055 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:32.168060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:32.168452 (+ 392us) server_negotiation.cc:407] Connection header received
0620 02:29:32.168550 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:32.168553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:32.168617 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:32.168709 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:32.169564 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.170317 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.171537 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.171954 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.172050 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:32.176487 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60431: BlockingRecv error: recv got EOF from 127.0.97.3:60431 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 02:29:32 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:32 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:32.203136 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:32.194251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:54413)
0620 02:29:32.194794 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:32.194812 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:32.194980 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:32.195359 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:32.195363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:32.195775 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:32.195976 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:32.195985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.197199 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.197203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:32.198108 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:32.198120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.199188 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.199195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:32.199400 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:32.200004 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:32.200023 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:32.202908 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":317,"threads_started":1}
W20260620 02:29:32.203409 9384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:32.194452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54413 (local address 127.0.97.62:39433)
0620 02:29:32.194717 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:32.194723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:32.194962 (+ 239us) server_negotiation.cc:407] Connection header received
0620 02:29:32.195099 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:32.195103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:32.195169 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:32.195263 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:32.196139 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.196940 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.198255 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.198921 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.199022 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:32.203137 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54413: BlockingRecv error: recv got EOF from 127.0.97.1:54413 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 02:29:32 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:32 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:32.714417 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:32.705124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:39347)
0620 02:29:32.705656 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:32.705675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:32.705820 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:32.706292 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:32.706296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:32.706703 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:32.706898 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:32.706907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.708157 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.708163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:32.709211 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:32.709225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.710013 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.710018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:32.710283 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:32.711042 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:32.711062 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:32.714165 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
W20260620 02:29:32.714956 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:32.705156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39347 (local address 127.0.97.62:39433)
0620 02:29:32.705981 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:32.705986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:32.706004 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:32.706056 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:32.706059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:32.706125 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:32.706222 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:32.707186 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.707985 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.709365 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:32.709826 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:32.709944 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:32.714459 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39347: BlockingRecv error: recv got EOF from 127.0.97.2:39347 (error 108)
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":643,"threads_started":1}
Jun 20 02:29:33 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:33 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:33.186826 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:33.177846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:59151)
0620 02:29:33.178420 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:33.178437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:33.178605 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:33.178981 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:33.178986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:33.179420 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:33.179574 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:33.179582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.180632 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.180640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:33.181624 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:33.181637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.182409 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.182415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:33.182596 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:33.183250 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:33.183268 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:33.186601 (+ 3333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":338,"threads_started":1}
W20260620 02:29:33.187024 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:33.177954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59151 (local address 127.0.97.62:39433)
0620 02:29:33.178217 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:33.178225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:33.178530 (+ 305us) server_negotiation.cc:407] Connection header received
0620 02:29:33.178742 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:33.178746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:33.178837 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:33.178949 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:33.179722 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.180487 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.181787 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.182235 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.182395 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:33.186809 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59151: BlockingRecv error: recv got EOF from 127.0.97.3:59151 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 02:29:33 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:33 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:33.213653 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:33.204278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:45783)
0620 02:29:33.205073 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:33.205097 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:33.205273 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:33.205681 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:33.205702 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:33.206144 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:33.206322 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:33.206331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.207346 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.207351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:33.208349 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:33.208361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.209174 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.209181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:33.209374 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:33.210053 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:33.210119 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:33.213397 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":300,"threads_started":1}
W20260620 02:29:33.213904 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:33.204442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45783 (local address 127.0.97.62:39433)
0620 02:29:33.204749 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:33.204755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:33.205219 (+ 464us) server_negotiation.cc:407] Connection header received
0620 02:29:33.205427 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:33.205435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:33.205517 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:33.205647 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:33.206472 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.207195 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.208499 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.208972 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.209158 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:33.213599 (+ 4441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45783: BlockingRecv error: recv got EOF from 127.0.97.1:45783 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Jun 20 02:29:33 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:33 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:33.724529 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:33.715455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:42409)
0620 02:29:33.716004 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:33.716022 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:33.716156 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:33.716685 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:33.716693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:33.717186 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:33.717358 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:33.717368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.718494 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.718499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:33.719441 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:33.719453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.720155 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.720164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:33.720311 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:33.721114 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:33.721133 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:33.724274 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":231,"threads_started":1}
W20260620 02:29:33.724766 9671 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:33.715537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42409 (local address 127.0.97.62:39433)
0620 02:29:33.716338 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:33.716343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:33.716359 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:29:33.716411 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:33.716415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:33.716489 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:33.716581 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:33.717495 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.718358 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.719577 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:33.719993 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:33.720092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:33.724482 (+ 4390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42409: BlockingRecv error: recv got EOF from 127.0.97.2:42409 (error 108)
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":581,"threads_started":1}
Jun 20 02:29:34 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:34 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:34.197268 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:34.187941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:37087)
0620 02:29:34.188636 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:34.188655 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:34.188842 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:34.189281 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:34.189286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:34.189767 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:34.189945 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:34.189954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.191151 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.191158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:34.192296 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:34.192308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.193115 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.193121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:34.193291 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:34.193936 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:34.193955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:34.196854 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":309,"threads_started":1}
W20260620 02:29:34.197386 9671 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:34.188144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37087 (local address 127.0.97.62:39433)
0620 02:29:34.188405 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:34.188412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:34.188752 (+ 340us) server_negotiation.cc:407] Connection header received
0620 02:29:34.188990 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:34.189020 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:34.189087 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:34.189240 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:34.190111 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.190989 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.192435 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.192910 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.193101 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:34.197133 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37087: BlockingRecv error: recv got EOF from 127.0.97.3:37087 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 02:29:34 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:34 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:34.224146 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:34.214654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:35739)
0620 02:29:34.215218 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:34.215240 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:34.215395 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:34.215785 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:34.215790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:34.216332 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:34.216586 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:34.216606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.217829 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.217835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:34.218841 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:34.218854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.219636 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.219642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:34.219816 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:34.220441 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:34.220462 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:34.223714 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":284,"threads_started":1}
W20260620 02:29:34.224298 9671 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:34.214913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35739 (local address 127.0.97.62:39433)
0620 02:29:34.215178 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:34.215183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:34.215325 (+ 142us) server_negotiation.cc:407] Connection header received
0620 02:29:34.215523 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:34.215529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:34.215634 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:34.215750 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:34.216738 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.217686 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.218992 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.219440 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.219588 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:34.223998 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35739: BlockingRecv error: recv got EOF from 127.0.97.1:35739 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 02:29:34 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:34 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:34.734799 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:34.725587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:40793)
0620 02:29:34.726117 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:34.726134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:34.726264 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:34.726623 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:34.726626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:34.727008 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:34.727157 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:34.727166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.728446 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.728449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:34.729449 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:34.729462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.730239 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.730244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:34.730384 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:34.731019 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:34.731069 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:34.734433 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
W20260620 02:29:34.734934 9675 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:34.725672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40793 (local address 127.0.97.62:39433)
0620 02:29:34.726193 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:34.726198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:34.726218 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:29:34.726406 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:34.726410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:34.726493 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:34.726586 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:34.727339 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.728308 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.729588 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:34.730034 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:34.730160 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:34.734682 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40793: BlockingRecv error: recv got EOF from 127.0.97.2:40793 (error 108)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
Jun 20 02:29:35 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:35 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:35.206933 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:35.198389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:34339)
0620 02:29:35.198868 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:35.198897 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:35.199017 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:35.199313 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:35.199317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:35.199721 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:35.199875 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:35.199883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.200877 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.200881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:35.201908 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:35.201920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.202732 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.202739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:35.202921 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:35.203684 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:35.203708 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:35.206688 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
W20260620 02:29:35.207127 9675 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:35.198467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34339 (local address 127.0.97.62:39433)
0620 02:29:35.198727 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:35.198734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:35.198980 (+ 246us) server_negotiation.cc:407] Connection header received
0620 02:29:35.199118 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:35.199122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:35.199185 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:35.199267 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:35.199994 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.200748 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.202042 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.202512 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.202653 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:35.206890 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34339: BlockingRecv error: recv got EOF from 127.0.97.3:34339 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 02:29:35 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:35 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:35.234133 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:35.225299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:51725)
0620 02:29:35.225793 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:35.225809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:35.225929 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:35.226241 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:35.226244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:35.226663 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:35.226910 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:35.226923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.228303 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.228309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:35.229471 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:35.229483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.230157 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.230161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:35.230275 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:35.230837 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:35.230864 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:35.233785 (+ 2921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
W20260620 02:29:35.234323 9675 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:35.225401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51725 (local address 127.0.97.62:39433)
0620 02:29:35.225649 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:35.225655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:35.225888 (+ 233us) server_negotiation.cc:407] Connection header received
0620 02:29:35.226029 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:35.226033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:35.226111 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:35.226220 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:35.227057 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.228151 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.229597 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.230024 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.230121 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:35.233993 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51725: BlockingRecv error: recv got EOF from 127.0.97.1:51725 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 02:29:35 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:35 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:35.744143 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:35.735884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:40727)
0620 02:29:35.736378 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:35.736394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:35.736499 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:35.736953 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:35.736956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:35.737418 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:35.737581 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:35.737589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.738655 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.738659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:35.739585 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:35.739595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.740308 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.740312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:35.740431 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:35.741101 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:35.741120 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:35.743894 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
W20260620 02:29:35.744374 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:35.735926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40727 (local address 127.0.97.62:39433)
0620 02:29:35.736691 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:35.736696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:35.736712 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:29:35.736754 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:35.736757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:35.736822 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:35.736910 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:35.737744 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.738520 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.739742 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:35.740175 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:35.740275 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:35.744110 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40727: BlockingRecv error: recv got EOF from 127.0.97.2:40727 (error 108)
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":581,"threads_started":1}
Jun 20 02:29:36 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:36 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:36.216548 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:36.207979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:53175)
0620 02:29:36.208688 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:36.208716 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:36.208886 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:36.209341 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:36.209348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:36.209753 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:36.210014 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:36.210026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.211085 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.211089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:36.212019 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:36.212029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.212664 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.212668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:36.212799 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:36.213374 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:36.213392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:36.216303 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":424,"threads_started":1}
W20260620 02:29:36.216750 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:36.208073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53175 (local address 127.0.97.62:39433)
0620 02:29:36.208258 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:36.208265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:36.208823 (+ 558us) server_negotiation.cc:407] Connection header received
0620 02:29:36.209048 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:36.209054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:36.209141 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:36.209262 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:36.210163 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.210949 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.212138 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.212536 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.212629 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:36.216506 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53175: BlockingRecv error: recv got EOF from 127.0.97.3:53175 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 02:29:36 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:36 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:36.243505 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:36.235102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:54973)
0620 02:29:36.235649 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:36.235665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:36.235776 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:36.236104 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:36.236107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:36.236480 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:36.236625 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:36.236634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.237838 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.237842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:36.238771 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:36.238782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.239481 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.239485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:36.239611 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:36.240289 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:36.240309 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:36.243272 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":328,"threads_started":1}
W20260620 02:29:36.243875 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:36.235187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54973 (local address 127.0.97.62:39433)
0620 02:29:36.235370 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:36.235375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:36.235742 (+ 367us) server_negotiation.cc:407] Connection header received
0620 02:29:36.235867 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:36.235870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:36.235974 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:36.236062 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:36.236829 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.237706 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.238925 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.239347 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.239444 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:36.243507 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54973: BlockingRecv error: recv got EOF from 127.0.97.1:54973 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:29:36 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:36 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:36.759507 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:36.745273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:54865)
0620 02:29:36.745845 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:36.745863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:36.745981 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:36.746298 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:36.746302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:36.746736 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:36.746880 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:36.746889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.748062 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.748067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:36.749171 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:36.749192 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.750224 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.750235 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:36.750357 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:36.756293 (+ 5936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:36.756314 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:36.759280 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":317,"threads_started":1}
W20260620 02:29:36.759856 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:36.745305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54865 (local address 127.0.97.62:39433)
0620 02:29:36.745852 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:36.745856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:36.745953 (+ 97us) server_negotiation.cc:407] Connection header received
0620 02:29:36.746080 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:36.746084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:36.746161 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:36.746252 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:36.747094 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.747889 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.749313 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:36.750047 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:36.750163 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:36.759575 (+ 9412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54865: BlockingRecv error: recv got EOF from 127.0.97.2:54865 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":327,"threads_started":1}
Jun 20 02:29:37 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:37 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:37.226001 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:37.217444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:51735)
0620 02:29:37.218095 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:37.218113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:37.218227 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:37.218562 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:37.218565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:37.218966 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:37.219150 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:37.219172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.220196 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.220200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:37.221198 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:37.221211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.221949 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.221953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:37.222077 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:37.222716 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:37.222735 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:37.225674 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":413,"threads_started":1}
W20260620 02:29:37.226166 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:37.217704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51735 (local address 127.0.97.62:39433)
0620 02:29:37.217889 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:37.217894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:37.218203 (+ 309us) server_negotiation.cc:407] Connection header received
0620 02:29:37.218346 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:37.218353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:37.218432 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:37.218519 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:37.219298 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.220076 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.221362 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.221811 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.221920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:37.225916 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51735: BlockingRecv error: recv got EOF from 127.0.97.3:51735 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 02:29:37 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:37 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:37.253611 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:37.244537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:39413)
0620 02:29:37.245143 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:37.245166 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:37.245334 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:37.245759 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:37.245764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:37.246209 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:37.246395 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:37.246404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.247551 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.247559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:37.248465 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:37.248477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.249288 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.249295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:37.249445 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:37.250048 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:37.250071 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:37.253376 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":293,"threads_started":1}
W20260620 02:29:37.253917 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:37.244611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39413 (local address 127.0.97.62:39433)
0620 02:29:37.244822 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:37.244830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:37.245262 (+ 432us) server_negotiation.cc:407] Connection header received
0620 02:29:37.245482 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:37.245486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:37.245592 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:37.245728 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:37.246541 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.247410 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.248622 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.249075 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.249210 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:37.253631 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39413: BlockingRecv error: recv got EOF from 127.0.97.1:39413 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 02:29:37 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:37 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:37.769286 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:37.760498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:57983)
0620 02:29:37.761207 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:37.761225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:37.761418 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:37.761817 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:37.761822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:37.762272 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:37.762466 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:37.762474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.763527 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.763531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:37.764451 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:37.764463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.765287 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.765292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:37.765418 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:37.766033 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:37.766053 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:37.768979 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":295,"threads_started":1}
W20260620 02:29:37.769520 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:37.760687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57983 (local address 127.0.97.62:39433)
0620 02:29:37.761232 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:37.761239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:37.761331 (+ 92us) server_negotiation.cc:407] Connection header received
0620 02:29:37.761552 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:37.761556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:37.761635 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:37.761737 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:37.762621 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.763383 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.764584 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:37.765150 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:37.765247 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:37.769250 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57983: BlockingRecv error: recv got EOF from 127.0.97.2:57983 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":312,"threads_started":1}
Jun 20 02:29:38 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:38 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:38.235927 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:38.226943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:38951)
0620 02:29:38.227555 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:38.227572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:38.227689 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:38.228071 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:38.228076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:38.228543 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:38.228718 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:38.228726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.229862 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.229874 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:38.230810 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:38.230822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.231553 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.231560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:38.231748 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:38.232422 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:38.232441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:38.235670 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":375,"threads_started":1}
W20260620 02:29:38.236121 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:38.227051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38951 (local address 127.0.97.62:39433)
0620 02:29:38.227281 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:38.227285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:38.227663 (+ 378us) server_negotiation.cc:407] Connection header received
0620 02:29:38.227851 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:38.227854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:38.227916 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:38.228030 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:38.228876 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.229709 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.230963 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.231387 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.231520 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:38.235880 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38951: BlockingRecv error: recv got EOF from 127.0.97.3:38951 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 02:29:38 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:38 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:38.263278 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:38.254568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:46449)
0620 02:29:38.255119 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:38.255135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:38.255250 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:38.255583 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:38.255586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:38.255971 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:38.256123 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:38.256131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.257211 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.257215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:38.258119 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:38.258129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.258973 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.258978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:38.259120 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:38.259787 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:38.259807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:38.262936 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":318,"threads_started":1}
W20260620 02:29:38.263622 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:38.254673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46449 (local address 127.0.97.62:39433)
0620 02:29:38.254846 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:38.254851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:38.255211 (+ 360us) server_negotiation.cc:407] Connection header received
0620 02:29:38.255347 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:38.255351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:38.255426 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:38.255521 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:38.256286 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.257088 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.258282 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.258805 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.258924 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:38.263164 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46449: BlockingRecv error: recv got EOF from 127.0.97.1:46449 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:29:38 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:38 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:38.779074 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:38.770208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:41345)
0620 02:29:38.770849 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:38.770867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:38.770982 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:38.771324 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:38.771327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:38.771859 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:38.772071 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:38.772081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.773560 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.773566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:38.774494 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:38.774506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.775213 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.775218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:38.775365 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:38.776072 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:38.776090 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:38.778840 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":298,"threads_started":1}
W20260620 02:29:38.779361 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:38.770402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41345 (local address 127.0.97.62:39433)
0620 02:29:38.770901 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:38.770906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:38.770924 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:29:38.771102 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:38.771105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:38.771175 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:38.771267 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:38.772208 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.773412 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.774626 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:38.775052 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:38.775151 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:38.779051 (+ 3900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41345: BlockingRecv error: recv got EOF from 127.0.97.2:41345 (error 108)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":285,"threads_started":1}
Jun 20 02:29:39 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:39 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:39.245280 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:39.236877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:41561)
0620 02:29:39.237522 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:39.237542 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:39.237703 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:39.238069 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:39.238074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:39.238511 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:39.238664 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:39.238672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.239685 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.239693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:39.240605 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:39.240616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.241369 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.241374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:39.241564 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:39.242143 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:39.242161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:39.244914 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":330,"threads_started":1}
W20260620 02:29:39.245386 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:39.236935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41561 (local address 127.0.97.62:39433)
0620 02:29:39.237178 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:39.237187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:39.237639 (+ 452us) server_negotiation.cc:407] Connection header received
0620 02:29:39.237864 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:39.237868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:39.237928 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:39.238037 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:39.238821 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.239541 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.240749 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.241221 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.241359 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:39.245154 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41561: BlockingRecv error: recv got EOF from 127.0.97.3:41561 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 02:29:39 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:39 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:39.272699 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:39.264332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:39841)
0620 02:29:39.264837 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:39.264853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:39.264968 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:39.265356 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:39.265360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:39.265758 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:39.265941 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:39.265949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.267028 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.267033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:39.267968 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:39.267979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.268683 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.268689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:39.268823 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:39.269524 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:39.269542 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:39.272389 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":270,"threads_started":1}
W20260620 02:29:39.272909 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:39.264380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39841 (local address 127.0.97.62:39433)
0620 02:29:39.264648 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:39.264655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:39.264932 (+ 277us) server_negotiation.cc:407] Connection header received
0620 02:29:39.265132 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:39.265136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:39.265210 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:39.265300 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:39.266092 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.266888 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.268095 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.268491 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.268584 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:39.272624 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39841: BlockingRecv error: recv got EOF from 127.0.97.1:39841 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 02:29:39 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:39 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:39.789119 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:39.780048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:32921)
0620 02:29:39.780609 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:39.780627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:39.780738 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:39.781280 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:39.781284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:39.781674 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:39.781861 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:39.781898 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.782981 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.782985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:39.784087 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:39.784100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.785136 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.785149 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:39.785278 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:39.785975 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:39.785994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:39.788807 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":290,"threads_started":1}
W20260620 02:29:39.789355 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:39.780314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32921 (local address 127.0.97.62:39433)
0620 02:29:39.780874 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:39.780878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:39.780895 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:29:39.780945 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:39.780950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:39.781048 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:39.781139 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:39.782037 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.782814 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.784225 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:39.784863 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:39.785052 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:39.789089 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32921: BlockingRecv error: recv got EOF from 127.0.97.2:32921 (error 108)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":354,"threads_started":1}
Jun 20 02:29:40 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:40 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:40.254825 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:40.246383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:56087)
0620 02:29:40.246981 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:40.247000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:40.247147 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:40.247478 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:40.247482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:40.247895 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:40.248066 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:40.248075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.249181 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.249186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:40.250148 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:40.250162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.250878 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.250882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:40.251001 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:40.251621 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:40.251639 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:40.254569 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":282,"threads_started":1}
W20260620 02:29:40.255036 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:40.246417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56087 (local address 127.0.97.62:39433)
0620 02:29:40.246596 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:40.246601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:40.247081 (+ 480us) server_negotiation.cc:407] Connection header received
0620 02:29:40.247254 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:40.247258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:40.247328 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:40.247412 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:40.248226 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.248970 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.250286 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.250719 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.250812 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:40.254782 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56087: BlockingRecv error: recv got EOF from 127.0.97.3:56087 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 02:29:40 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:40 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:40.282644 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:40.273830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:43175)
0620 02:29:40.274372 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:40.274392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:40.274528 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:40.274951 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:40.274955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:40.275363 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:40.275525 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:40.275535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.276601 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.276606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:40.277572 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:40.277584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.278272 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.278277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:40.278402 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:40.279108 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:40.279127 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:40.282390 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":268,"threads_started":1}
W20260620 02:29:40.282829 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:40.273955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43175 (local address 127.0.97.62:39433)
0620 02:29:40.274374 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:40.274379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:40.274485 (+ 106us) server_negotiation.cc:407] Connection header received
0620 02:29:40.274678 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:40.274689 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:40.274781 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:40.274881 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:40.275691 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.276464 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.277711 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.278131 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.278234 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:40.282576 (+ 4342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43175: BlockingRecv error: recv got EOF from 127.0.97.1:43175 (error 108)
Metrics: {"server-negotiator.queue_time_us":245}
Jun 20 02:29:40 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:40 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:40.798770 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:40.790139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:45685)
0620 02:29:40.790725 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:40.790743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:40.790849 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:40.791198 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:40.791201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:40.791607 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:40.791767 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:40.791776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.792803 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.792807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:40.793739 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:40.793750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.794474 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.794478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:40.794604 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:40.795310 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:40.795328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:40.798412 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":351,"threads_started":1}
W20260620 02:29:40.798890 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:40.790313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45685 (local address 127.0.97.62:39433)
0620 02:29:40.790765 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:40.790769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:40.790794 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:29:40.790955 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:40.790959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:40.791052 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:40.791151 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:40.791902 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.792660 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.793878 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:40.794314 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:40.794418 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:40.798636 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45685: BlockingRecv error: recv got EOF from 127.0.97.2:45685 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
Jun 20 02:29:41 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:41 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:41.265450 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:41.255864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.3:58893)
0620 02:29:41.256412 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:41.256431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:41.256616 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:41.257045 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:41.257050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:41.257506 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:41.257708 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:41.257719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.258833 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.258839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:41.259813 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:41.259826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.260597 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.260604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:41.260835 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:41.261664 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:41.261694 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:41.265006 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
W20260620 02:29:41.265502 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:41.255920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58893 (local address 127.0.97.62:39433)
0620 02:29:41.256210 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:41.256218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:41.256542 (+ 324us) server_negotiation.cc:407] Connection header received
0620 02:29:41.256774 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:41.256782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:41.256858 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:41.256981 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:41.257880 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.258669 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.259975 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.260429 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.260592 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:41.265258 (+ 4666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58893: BlockingRecv error: recv got EOF from 127.0.97.3:58893 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jun 20 02:29:41 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:41 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:41.292752 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:41.283995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.1:44805)
0620 02:29:41.284533 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:41.284549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:41.284717 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:41.285110 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:41.285115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:41.285546 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:41.285805 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:41.285819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.287071 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.287076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:41.288071 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:41.288083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.288784 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.288790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:41.288961 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:41.289589 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:41.289607 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:41.292380 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":315,"threads_started":1}
W20260620 02:29:41.292866 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:41.284114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44805 (local address 127.0.97.62:39433)
0620 02:29:41.284352 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:41.284361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:41.284636 (+ 275us) server_negotiation.cc:407] Connection header received
0620 02:29:41.284858 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:41.284862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:41.284933 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:41.285062 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:41.285950 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.286930 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.288215 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.288629 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.288762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:41.292593 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44805: BlockingRecv error: recv got EOF from 127.0.97.1:44805 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 02:29:41 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:29:41 dist-test-slave-r706 krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:29:41.813309 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:41.800151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39433 (local address 127.0.97.2:36475)
0620 02:29:41.800725 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:41.800743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:41.800855 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:41.801395 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:41.801398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:41.801829 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:41.802010 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:41.802020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.803225 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.803230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:41.804166 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:41.804178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.805074 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.805080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:41.805207 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:41.809985 (+ 4778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:41.810009 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:41.812973 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39433: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":334,"threads_started":1}
W20260620 02:29:41.813714 9703 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:41.800374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36475 (local address 127.0.97.62:39433)
0620 02:29:41.801047 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:41.801057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:41.801089 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:29:41.801152 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:41.801156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:41.801231 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:41.801352 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:41.802247 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.803044 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.804317 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:41.804851 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:41.805047 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:41.813383 (+ 8336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36475: BlockingRecv error: recv got EOF from 127.0.97.2:36475 (error 108)
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":370,"threads_started":1}
I20260620 02:29:42.091423 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9235
I20260620 02:29:42.104669 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:42.092747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.97.1:40347)
0620 02:29:42.093377 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:42.093384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:42.093412 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:42.093479 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:42.093486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:42.093699 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:42.093863 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:42.095317 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.096396 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.097713 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.098355 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.098494 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:42.101210 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:42.101220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:42.101241 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:42.101285 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:42.104109 (+ 2824us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:29:42.104417 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51390: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":323,"threads_started":1}
W20260620 02:29:42.104846 9705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51390: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:29:42.105096 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9235: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40347: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:29:42.105412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9235
I20260620 02:29:42.123581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9386
I20260620 02:29:42.137200 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:42.124797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.0.97.2:38265)
0620 02:29:42.125420 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:42.125428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:42.125458 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:29:42.125515 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:42.125522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:42.125760 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:42.125963 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:42.127317 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.128406 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.129736 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.130321 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.130466 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:42.133505 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:42.133516 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:42.133530 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:42.133607 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:42.136423 (+ 2816us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:29:42.136795 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47612: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":327,"threads_started":1}
W20260620 02:29:42.137311 9706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47612: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:29:42.137537 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9386: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38265: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:29:42.137745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9386
I20260620 02:29:42.155097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9517
I20260620 02:29:42.167769 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:42.156182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.0.97.3:37745)
0620 02:29:42.156816 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:42.156823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:42.156851 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:42.156921 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:42.156927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:42.157223 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:42.157381 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:42.158660 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.159727 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.160874 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.161574 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.161678 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:42.164200 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:42.164211 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:42.164226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:42.164270 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:42.167152 (+ 2882us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:29:42.167450 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35536: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":334,"threads_started":1}
W20260620 02:29:42.167891 9707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35536: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:29:42.167933 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9517: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:37745: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:29:42.168134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9517
I20260620 02:29:42.184948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9166
I20260620 02:29:42.186501 9228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:42.307426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9166
2026-06-20T02:29:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21585 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T02:29:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:42Z Disabled control of system clock
I20260620 02:29:42.393471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35043
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:29:42.732658 9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:42.733135 9712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:42.733322 9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:42.741725 9712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:42.741864 9712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:42.741961 9712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:42.742070 9712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:42.742162 9712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:42.752199 9712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:42.753790 9712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:42.755777 9712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:42.769258 9718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:42.769263 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:42.770057 9712 server_base.cc:1034] running on GCE node
W20260620 02:29:42.770190 9717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:42.771227 9712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:42.772830 9712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:42.774106 9712 hybrid_clock.cc:648] HybridClock initialized: now 1781922582774071 us; error 89 us; skew 500 ppm
I20260620 02:29:42.774531 9712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:42.777153 9712 webserver.cc:460] Webserver started at http://127.0.97.62:39755/ using document root <none> and password file <none>
I20260620 02:29:42.777905 9712 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:42.778049 9712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:42.778396 9712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:42.781463 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "2770e9f28b8148b8987a6869d8d6bdc8"
format_stamp: "Formatted at 2026-06-20 02:29:42 on dist-test-slave-r706"
server_key: "ea551980d823f87142d2acbe3a368faf"
server_key_iv: "4322ced79f7101db17027c7ff68f736a"
server_key_version: "encryptionkey@0"
I20260620 02:29:42.782271 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "2770e9f28b8148b8987a6869d8d6bdc8"
format_stamp: "Formatted at 2026-06-20 02:29:42 on dist-test-slave-r706"
server_key: "ea551980d823f87142d2acbe3a368faf"
server_key_iv: "4322ced79f7101db17027c7ff68f736a"
server_key_version: "encryptionkey@0"
I20260620 02:29:42.787760 9712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:29:42.791538 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:42.792994 9712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:29:42.793298 9712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "2770e9f28b8148b8987a6869d8d6bdc8"
format_stamp: "Formatted at 2026-06-20 02:29:42 on dist-test-slave-r706"
server_key: "ea551980d823f87142d2acbe3a368faf"
server_key_iv: "4322ced79f7101db17027c7ff68f736a"
server_key_version: "encryptionkey@0"
I20260620 02:29:42.793571 9712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:42.824286 9712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:42.825313 9712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:42.825647 9712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:42.844518 9712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35043
I20260620 02:29:42.844527 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35043 every 8 connection(s)
I20260620 02:29:42.845948 9712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:42.849613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9712
I20260620 02:29:42.849820 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:29:42.849959 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:42.850543 388 external_mini_cluster.cc:1428] Setting key c07f33aaf209d25b68f88694101ca585
I20260620 02:29:42.865391 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8: Bootstrap starting.
I20260620 02:29:42.867974 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:42.852739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.97.62:35043)
0620 02:29:42.854030 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:42.854044 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:42.854103 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:29:42.857057 (+ 2954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:42.857088 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:42.857798 (+ 710us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:42.858282 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:42.858848 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.860738 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.862278 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:42.863411 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:42.863605 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:42.864855 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:42.864865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:42.864885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:42.865677 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:42.865797 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:42.865825 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:42.865956 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:42.866194 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 02:29:42.866554 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":473,"threads_started":1}
I20260620 02:29:42.869515 9778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:42.870630 9778 log.cc:824] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:42.873755 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8: No bootstrap required, opened a new log
I20260620 02:29:42.881842 9778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260620 02:29:42.882301 9778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:42.882411 9778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2770e9f28b8148b8987a6869d8d6bdc8, State: Initialized, Role: FOLLOWER
I20260620 02:29:42.883040 9778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260620 02:29:42.883281 9778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:42.883425 9778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:42.883607 9778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:42.886035 9778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260620 02:29:42.886602 9778 leader_election.cc:302] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2770e9f28b8148b8987a6869d8d6bdc8; no voters:
I20260620 02:29:42.887216 9778 leader_election.cc:288] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:29:42.887384 9783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:42.889144 9783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [term 1 LEADER]: Becoming Leader. State: Replica: 2770e9f28b8148b8987a6869d8d6bdc8, State: Running, Role: LEADER
I20260620 02:29:42.889757 9783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } }
I20260620 02:29:42.890461 9778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:42.892940 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2770e9f28b8148b8987a6869d8d6bdc8. Latest consensus state: current_term: 1 leader_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } } }
I20260620 02:29:42.893064 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2770e9f28b8148b8987a6869d8d6bdc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35043 } } }
I20260620 02:29:42.893445 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:42.893496 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:42.895709 9788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:42.902047 9788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:42.911576 9788 catalog_manager.cc:1347] Generated new cluster ID: b73d4655e4894d2cb10b855627396379
I20260620 02:29:42.911713 9788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:42.923951 9788 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:29:42.925428 9788 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:42.934676 9788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2770e9f28b8148b8987a6869d8d6bdc8: Generated new TSK 0
I20260620 02:29:42.935578 9788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:29:42.942413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35043
--builtin_ntp_servers=127.0.97.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 02:29:43.291194 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:43.291600 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:43.291708 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:43.299571 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:43.299692 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:43.299809 9802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:43.309290 9802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:43.310556 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:43.312296 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:43.327023 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:43.326911 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:43.326947 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:43.328395 9802 server_base.cc:1034] running on GCE node
I20260620 02:29:43.329314 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:43.330662 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:43.331912 9802 hybrid_clock.cc:648] HybridClock initialized: now 1781922583331798 us; error 103 us; skew 500 ppm
I20260620 02:29:43.332249 9802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:43.334632 9802 webserver.cc:460] Webserver started at http://127.0.97.1:46877/ using document root <none> and password file <none>
I20260620 02:29:43.335346 9802 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:43.335438 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:43.335713 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:43.338732 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "c8350afdd05f4867ad8375137fdfbea8"
format_stamp: "Formatted at 2026-06-20 02:29:43 on dist-test-slave-r706"
server_key: "a689defe849db76446d2a0373b606c1d"
server_key_iv: "f02218d14879d6b9e7ab343a4e92157a"
server_key_version: "encryptionkey@0"
I20260620 02:29:43.339491 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c8350afdd05f4867ad8375137fdfbea8"
format_stamp: "Formatted at 2026-06-20 02:29:43 on dist-test-slave-r706"
server_key: "a689defe849db76446d2a0373b606c1d"
server_key_iv: "f02218d14879d6b9e7ab343a4e92157a"
server_key_version: "encryptionkey@0"
I20260620 02:29:43.344799 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:29:43.348304 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:43.349849 9802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:29:43.350033 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "c8350afdd05f4867ad8375137fdfbea8"
format_stamp: "Formatted at 2026-06-20 02:29:43 on dist-test-slave-r706"
server_key: "a689defe849db76446d2a0373b606c1d"
server_key_iv: "f02218d14879d6b9e7ab343a4e92157a"
server_key_version: "encryptionkey@0"
I20260620 02:29:43.350258 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:43.375263 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:43.376170 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:43.376464 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:43.377491 9802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:43.378981 9802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:43.379068 9802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:43.379150 9802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:43.379204 9802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:43.413892 9802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43973
I20260620 02:29:43.414011 9928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43973 every 8 connection(s)
I20260620 02:29:43.415345 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:43.425421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9802
I20260620 02:29:43.425649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:29:43.426312 388 external_mini_cluster.cc:1428] Setting key 8ca3f4d4aeb79d4e6cf88a1d114a4637
I20260620 02:29:43.428408 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:43.418152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33477 (local address 127.0.97.62:35043)
0620 02:29:43.418632 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:43.418637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:43.419739 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 02:29:43.421668 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:43.421674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:43.421770 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:43.421903 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:43.422988 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.423738 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.425377 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.425885 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.426018 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:43.427548 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:43.427561 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:43.427564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:43.427582 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:43.427619 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:43.427632 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:43.427731 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:43.427953 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 02:29:43.428051 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
I20260620 02:29:43.429311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35043
--builtin_ntp_servers=127.0.97.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:29:43.429551 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:43.418626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35043 (local address 127.0.97.1:33477)
0620 02:29:43.419581 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:43.419617 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:43.421349 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:43.422141 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:43.422149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:43.422247 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:43.422814 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:43.422831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.423912 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.423916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:43.425227 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:43.425240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.426126 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.426133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:43.426379 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:43.427180 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:29:43.427185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:43.427248 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:29:43.427343 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:43.427731 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:43.427745 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:43.427756 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:29:43.427944 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 02:29:43.428358 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":307,"threads_started":1}
I20260620 02:29:43.431226 9929 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35043
I20260620 02:29:43.431546 9929 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:43.432296 9929 heartbeater.cc:511] Master 127.0.97.62:35043 requested a full tablet report, sending...
I20260620 02:29:43.434688 9743 ts_manager.cc:194] Registered new tserver with Master: c8350afdd05f4867ad8375137fdfbea8 (127.0.97.1:43973)
I20260620 02:29:43.436969 9743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33477
W20260620 02:29:43.761327 9933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:43.761655 9933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:43.761823 9933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:43.769203 9933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:43.769347 9933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:43.769531 9933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:43.779431 9933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:43.780834 9933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:43.782644 9933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:43.796064 9939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:43.796070 9941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:43.797247 9938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:43.797569 9933 server_base.cc:1034] running on GCE node
I20260620 02:29:43.798357 9933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:43.799474 9933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:43.800730 9933 hybrid_clock.cc:648] HybridClock initialized: now 1781922583800679 us; error 47 us; skew 500 ppm
I20260620 02:29:43.801172 9933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:43.803438 9933 webserver.cc:460] Webserver started at http://127.0.97.2:36537/ using document root <none> and password file <none>
I20260620 02:29:43.804160 9933 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:43.804291 9933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:43.804641 9933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:43.807602 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "57013df321974250b778f9afbc4434a7"
format_stamp: "Formatted at 2026-06-20 02:29:43 on dist-test-slave-r706"
server_key: "2f9b32be34312c9d244c5bcaa69b8284"
server_key_iv: "17e4f1c6e2031075a2a7f3b2f772ff8a"
server_key_version: "encryptionkey@0"
I20260620 02:29:43.808409 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "57013df321974250b778f9afbc4434a7"
format_stamp: "Formatted at 2026-06-20 02:29:43 on dist-test-slave-r706"
server_key: "2f9b32be34312c9d244c5bcaa69b8284"
server_key_iv: "17e4f1c6e2031075a2a7f3b2f772ff8a"
server_key_version: "encryptionkey@0"
I20260620 02:29:43.813736 9933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 02:29:43.817224 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:43.818545 9933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:29:43.818729 9933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "57013df321974250b778f9afbc4434a7"
format_stamp: "Formatted at 2026-06-20 02:29:43 on dist-test-slave-r706"
server_key: "2f9b32be34312c9d244c5bcaa69b8284"
server_key_iv: "17e4f1c6e2031075a2a7f3b2f772ff8a"
server_key_version: "encryptionkey@0"
I20260620 02:29:43.818953 9933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:43.834394 9933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:43.835287 9933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:43.835593 9933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:43.836637 9933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:43.838132 9933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:43.838256 9933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:43.838395 9933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:43.838483 9933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:43.873162 9933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36083
I20260620 02:29:43.873214 10059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36083 every 8 connection(s)
I20260620 02:29:43.874476 9933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:43.877240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 9933
I20260620 02:29:43.877431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:29:43.877975 388 external_mini_cluster.cc:1428] Setting key 05b118941e1b06b70e6671e08cb1a8ae
I20260620 02:29:43.881254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35043
--builtin_ntp_servers=127.0.97.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:29:43.888679 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:43.877234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53125 (local address 127.0.97.62:35043)
0620 02:29:43.877430 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:43.877435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:43.879303 (+ 1868us) server_negotiation.cc:407] Connection header received
0620 02:29:43.881482 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:43.881491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:43.881585 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:43.881693 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:43.882733 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.883919 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.885548 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.886236 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.886386 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:43.887726 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:43.887735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:43.887742 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:43.887769 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:43.887816 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:43.887836 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:43.887937 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:43.888139 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 02:29:43.888268 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:29:43.889591 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:43.878034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35043 (local address 127.0.97.2:53125)
0620 02:29:43.879108 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:43.879186 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:43.881157 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:43.881893 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:43.881902 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:43.881987 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:43.882558 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:43.882573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.884064 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.884068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:43.885405 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:43.885416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:43.886442 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:43.886446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:43.886642 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:43.887381 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:29:43.887384 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:43.887424 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:29:43.887505 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:43.887939 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:43.887946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:43.887951 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:29:43.888107 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:29:43.888408 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":328,"threads_started":1}
I20260620 02:29:43.891158 10060 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35043
I20260620 02:29:43.891525 10060 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:43.892439 10060 heartbeater.cc:511] Master 127.0.97.62:35043 requested a full tablet report, sending...
I20260620 02:29:43.894151 9743 ts_manager.cc:194] Registered new tserver with Master: 57013df321974250b778f9afbc4434a7 (127.0.97.2:36083)
I20260620 02:29:43.895005 9743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53125
W20260620 02:29:44.218333 10063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:44.218725 10063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:44.218914 10063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:44.226845 10063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:44.227026 10063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:44.227217 10063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:44.236616 10063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:44.238086 10063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:44.239948 10063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:44.253947 10069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:44.253984 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:44.254027 10068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:44.259289 10063 server_base.cc:1034] running on GCE node
I20260620 02:29:44.259982 10063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:44.261090 10063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:44.262322 10063 hybrid_clock.cc:648] HybridClock initialized: now 1781922584262264 us; error 70 us; skew 500 ppm
I20260620 02:29:44.262636 10063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:44.264925 10063 webserver.cc:460] Webserver started at http://127.0.97.3:33767/ using document root <none> and password file <none>
I20260620 02:29:44.265643 10063 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:44.265786 10063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:44.266134 10063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:44.269120 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "cbf589c6d3bf40e1a348c17a31e4957a"
format_stamp: "Formatted at 2026-06-20 02:29:44 on dist-test-slave-r706"
server_key: "b4b3f0a0618ee02342302073b80e9278"
server_key_iv: "6ab5f567fadd5e712d4543f58b2f721c"
server_key_version: "encryptionkey@0"
I20260620 02:29:44.269920 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cbf589c6d3bf40e1a348c17a31e4957a"
format_stamp: "Formatted at 2026-06-20 02:29:44 on dist-test-slave-r706"
server_key: "b4b3f0a0618ee02342302073b80e9278"
server_key_iv: "6ab5f567fadd5e712d4543f58b2f721c"
server_key_version: "encryptionkey@0"
I20260620 02:29:44.275220 10063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:29:44.278941 10077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:44.280243 10063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:29:44.280426 10063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "cbf589c6d3bf40e1a348c17a31e4957a"
format_stamp: "Formatted at 2026-06-20 02:29:44 on dist-test-slave-r706"
server_key: "b4b3f0a0618ee02342302073b80e9278"
server_key_iv: "6ab5f567fadd5e712d4543f58b2f721c"
server_key_version: "encryptionkey@0"
I20260620 02:29:44.280654 10063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:44.300928 10063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:44.301856 10063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:44.302157 10063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:44.303169 10063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:44.304678 10063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:44.304819 10063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:44.304952 10063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:44.305081 10063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:29:44.341619 10063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40833
I20260620 02:29:44.341676 10189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40833 every 8 connection(s)
I20260620 02:29:44.342941 10063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:44.349972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10063
I20260620 02:29:44.350188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:29:44.350675 388 external_mini_cluster.cc:1428] Setting key 9e99da8a4ba4ca09681a0a599224b852
I20260620 02:29:44.355973 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:44.345634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37997 (local address 127.0.97.62:35043)
0620 02:29:44.345833 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:44.345838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:44.347217 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 02:29:44.348919 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:44.348924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:44.348994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:44.349158 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:44.350203 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.351067 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.352985 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.353657 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.353797 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:44.355124 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:44.355129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:44.355134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:44.355160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:44.355191 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:44.355202 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:44.355298 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:44.355552 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:29:44.355647 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:29:44.356989 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:44.346152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35043 (local address 127.0.97.3:37997)
0620 02:29:44.347034 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:44.347074 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:44.348644 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:44.349358 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:44.349366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:44.349448 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:44.350012 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:44.350028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.351213 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.351220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:44.352844 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:44.352856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.353899 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.353907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:44.354139 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:44.354808 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:29:44.354820 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:44.354858 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:29:44.354936 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:44.355295 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:44.355322 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:44.355333 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:29:44.355540 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 02:29:44.355936 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":275,"threads_started":1}
I20260620 02:29:44.358477 10190 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35043
I20260620 02:29:44.358801 10190 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:44.359591 10190 heartbeater.cc:511] Master 127.0.97.62:35043 requested a full tablet report, sending...
I20260620 02:29:44.360970 9743 ts_manager.cc:194] Registered new tserver with Master: cbf589c6d3bf40e1a348c17a31e4957a (127.0.97.3:40833)
I20260620 02:29:44.361806 9743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37997
I20260620 02:29:44.366288 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:29:44.372439 9931 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:44.370860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.0.97.62:35043)
0620 02:29:44.371061 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:44.371066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:44.371617 (+ 551us) server_negotiation.cc:407] Connection header received
0620 02:29:44.371756 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:44.371760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:44.371832 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:44.371915 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:44.372242 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44158: BlockingRecv error: recv got EOF from 127.0.0.1:44158 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:29:44.373070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9802
I20260620 02:29:44.380752 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:44.373997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.0.97.1:43973)
0620 02:29:44.374558 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:44.374565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:44.374593 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:44.374654 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:44.374661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:44.374870 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:44.374995 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:44.375326 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.376343 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.377659 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.378245 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.378373 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:44.379422 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:44.379430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:44.379444 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:44.380059 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:44.380105 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:44.380125 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:44.380233 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:44.380344 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 02:29:44.380451 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
I20260620 02:29:44.382074 9924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:44.464264 9929 heartbeater.cc:503] Master 127.0.97.62:35043 was elected leader, sending a full tablet report...
I20260620 02:29:44.465464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9802
W20260620 02:29:44.482326 9727 connection.cc:535] server connection from 127.0.97.1:33477 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:29:44.482692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9933
I20260620 02:29:44.490553 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:44.483509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.0.97.2:36083)
0620 02:29:44.484051 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:44.484058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:44.484085 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:29:44.484153 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:44.484160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:44.484370 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:44.484571 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:44.484938 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.485985 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.487296 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.487836 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.487961 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:44.489133 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:44.489150 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:44.489163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:44.489784 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:44.489834 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:44.489854 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:44.489950 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:44.490083 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 02:29:44.490243 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
I20260620 02:29:44.491855 10055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:44.576321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9933
I20260620 02:29:44.593329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10063
I20260620 02:29:44.602208 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:44.594414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.97.3:40833)
0620 02:29:44.595038 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:44.595045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:44.595075 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:29:44.595136 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:44.595142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:44.595365 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:44.595513 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:44.595935 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.597105 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.598591 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:44.599207 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:44.599308 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:44.600496 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:44.600506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:44.600519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:44.601312 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:44.601363 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:44.601384 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:44.601486 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:44.601656 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:29:44.601782 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":314,"threads_started":1}
I20260620 02:29:44.603590 10185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:44.686867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10063
I20260620 02:29:44.703640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 9712
I20260620 02:29:44.704901 9773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:44.809696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 9712
2026-06-20T02:29:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2494 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T02:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:44Z Disabled control of system clock
I20260620 02:29:44.888739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34865
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:29:45.244531 10209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:45.244876 10209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:45.244967 10209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:45.252905 10209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:45.253124 10209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:45.253180 10209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:45.253252 10209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:45.253324 10209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:45.262931 10209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:45.264407 10209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:45.266204 10209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:45.278771 10215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:45.278717 10217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:45.279352 10214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:45.279961 10209 server_base.cc:1034] running on GCE node
I20260620 02:29:45.280768 10209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:45.282388 10209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:45.283656 10209 hybrid_clock.cc:648] HybridClock initialized: now 1781922585283622 us; error 52 us; skew 500 ppm
I20260620 02:29:45.284054 10209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:45.286563 10209 webserver.cc:460] Webserver started at http://127.0.97.62:32989/ using document root <none> and password file <none>
I20260620 02:29:45.287345 10209 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:45.287489 10209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:45.287834 10209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:45.291075 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "3bd5fb0a28474895b5ea32035a03976f"
format_stamp: "Formatted at 2026-06-20 02:29:45 on dist-test-slave-r706"
server_key: "bdc3b058db78d73d81974f630663f684"
server_key_iv: "43a989214a2eb852cff92040996b6e2c"
server_key_version: "encryptionkey@0"
I20260620 02:29:45.291975 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "3bd5fb0a28474895b5ea32035a03976f"
format_stamp: "Formatted at 2026-06-20 02:29:45 on dist-test-slave-r706"
server_key: "bdc3b058db78d73d81974f630663f684"
server_key_iv: "43a989214a2eb852cff92040996b6e2c"
server_key_version: "encryptionkey@0"
I20260620 02:29:45.298058 10209 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:29:45.301900 10223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:45.303512 10209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:29:45.303778 10209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "3bd5fb0a28474895b5ea32035a03976f"
format_stamp: "Formatted at 2026-06-20 02:29:45 on dist-test-slave-r706"
server_key: "bdc3b058db78d73d81974f630663f684"
server_key_iv: "43a989214a2eb852cff92040996b6e2c"
server_key_version: "encryptionkey@0"
I20260620 02:29:45.304047 10209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:45.310282 10209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:45.311147 10209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:45.311478 10209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:45.332435 10209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34865
I20260620 02:29:45.332471 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34865 every 8 connection(s)
I20260620 02:29:45.333928 10209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:45.335453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10209
I20260620 02:29:45.335685 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:29:45.336548 388 external_mini_cluster.cc:1428] Setting key 97e99a72f152fd17abbd65492c49dcae
I20260620 02:29:45.339278 10275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:45.346977 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:45.338842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.0.97.62:34865)
0620 02:29:45.341389 (+ 2547us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:45.341401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:45.341444 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:29:45.343040 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:45.343065 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:45.343525 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:45.343976 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:45.345184 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:45.345191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:45.345204 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:45.345218 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:45.345301 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:45.345318 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:45.345581 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:45.345735 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:29:45.346012 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2026,"thread_start_us":1798,"threads_started":1}
I20260620 02:29:45.353065 10275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f: Bootstrap starting.
I20260620 02:29:45.356436 10275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:45.357668 10275 log.cc:824] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:45.360792 10275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f: No bootstrap required, opened a new log
I20260620 02:29:45.368580 10275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5fb0a28474895b5ea32035a03976f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260620 02:29:45.369081 10275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:45.369207 10275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd5fb0a28474895b5ea32035a03976f, State: Initialized, Role: FOLLOWER
I20260620 02:29:45.369820 10275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5fb0a28474895b5ea32035a03976f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260620 02:29:45.370079 10275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:45.370237 10275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:45.370432 10275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:45.372711 10275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5fb0a28474895b5ea32035a03976f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260620 02:29:45.373304 10275 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd5fb0a28474895b5ea32035a03976f; no voters:
I20260620 02:29:45.374119 10275 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:29:45.374358 10280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:45.375792 10280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [term 1 LEADER]: Becoming Leader. State: Replica: 3bd5fb0a28474895b5ea32035a03976f, State: Running, Role: LEADER
I20260620 02:29:45.376447 10280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5fb0a28474895b5ea32035a03976f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260620 02:29:45.376981 10275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:45.379770 10281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd5fb0a28474895b5ea32035a03976f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5fb0a28474895b5ea32035a03976f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } } }
I20260620 02:29:45.380172 10281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:45.379869 10282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd5fb0a28474895b5ea32035a03976f. Latest consensus state: current_term: 1 leader_uuid: "3bd5fb0a28474895b5ea32035a03976f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5fb0a28474895b5ea32035a03976f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } } }
I20260620 02:29:45.380333 10282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:45.385710 10286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:45.391620 10286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:45.400899 10286 catalog_manager.cc:1347] Generated new cluster ID: 64e20c310a474e41850502bd5233fd11
I20260620 02:29:45.401069 10286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:45.438457 10286 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:29:45.438995 10286 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:45.455991 10286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd5fb0a28474895b5ea32035a03976f: Generated new TSK 0
I20260620 02:29:45.456868 10286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:29:45.462411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34865
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:29:45.803020 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:45.803337 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:45.803488 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:45.811326 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:45.811445 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:45.811636 10299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:45.821239 10299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:45.822582 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:45.824422 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:45.838414 10307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:45.838369 10304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:45.838466 10305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:45.842828 10299 server_base.cc:1034] running on GCE node
I20260620 02:29:45.843483 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:45.844528 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:45.845741 10299 hybrid_clock.cc:648] HybridClock initialized: now 1781922585845697 us; error 37 us; skew 500 ppm
I20260620 02:29:45.846179 10299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:45.848696 10299 webserver.cc:460] Webserver started at http://127.0.97.1:39943/ using document root <none> and password file <none>
I20260620 02:29:45.849498 10299 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:45.849593 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:45.849848 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:45.852842 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "7f204d9ad69548ada8df11ae76afc15b"
format_stamp: "Formatted at 2026-06-20 02:29:45 on dist-test-slave-r706"
server_key: "45c56449cc1106a49a9131e298e0b760"
server_key_iv: "71e469b8e26c1c9bcbd56b76de693e66"
server_key_version: "encryptionkey@0"
I20260620 02:29:45.853663 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7f204d9ad69548ada8df11ae76afc15b"
format_stamp: "Formatted at 2026-06-20 02:29:45 on dist-test-slave-r706"
server_key: "45c56449cc1106a49a9131e298e0b760"
server_key_iv: "71e469b8e26c1c9bcbd56b76de693e66"
server_key_version: "encryptionkey@0"
I20260620 02:29:45.858901 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:29:45.862552 10313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:45.863947 10299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:29:45.864143 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "7f204d9ad69548ada8df11ae76afc15b"
format_stamp: "Formatted at 2026-06-20 02:29:45 on dist-test-slave-r706"
server_key: "45c56449cc1106a49a9131e298e0b760"
server_key_iv: "71e469b8e26c1c9bcbd56b76de693e66"
server_key_version: "encryptionkey@0"
I20260620 02:29:45.864379 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:45.870313 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:45.871129 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:45.871400 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:45.872399 10299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:45.874010 10299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:45.874132 10299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:45.874266 10299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:45.874362 10299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:45.910352 10299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34195
I20260620 02:29:45.910406 10425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34195 every 8 connection(s)
I20260620 02:29:45.911720 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:45.918370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10299
I20260620 02:29:45.918609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:29:45.919415 388 external_mini_cluster.cc:1428] Setting key 6fef4e63e63b2c8eb0bb1bc8b2ca9d4a
I20260620 02:29:45.920652 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:45.914516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55155 (local address 127.0.97.62:34865)
0620 02:29:45.915022 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:45.915026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:45.916083 (+ 1057us) server_negotiation.cc:407] Connection header received
0620 02:29:45.917900 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:45.917905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:45.917987 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:45.918116 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:45.919813 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:45.919820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:45.919829 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:45.919848 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:45.919893 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:45.919906 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:45.920061 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:45.920265 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 02:29:45.920338 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":293,"threads_started":1}
I20260620 02:29:45.921864 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:45.915002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34865 (local address 127.0.97.1:55155)
0620 02:29:45.915914 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:45.915950 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:45.917615 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:45.918369 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:45.918377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:45.918472 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:45.919405 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:29:45.919408 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:45.919448 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:29:45.919601 (+ 153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:45.920045 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:45.920054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:45.920060 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:29:45.920345 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 02:29:45.920623 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260620 02:29:45.922573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34865
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:29:45.923491 10426 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34865
I20260620 02:29:45.923810 10426 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:45.924402 10426 heartbeater.cc:511] Master 127.0.97.62:34865 requested a full tablet report, sending...
I20260620 02:29:45.926484 10240 ts_manager.cc:194] Registered new tserver with Master: 7f204d9ad69548ada8df11ae76afc15b (127.0.97.1:34195)
W20260620 02:29:46.261554 10430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:46.261858 10430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:46.261951 10430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:46.269588 10430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:46.269702 10430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:46.269812 10430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:46.279232 10430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:46.280503 10430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:46.282287 10430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:46.296001 10435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:46.295936 10436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:46.295969 10438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:46.296855 10430 server_base.cc:1034] running on GCE node
I20260620 02:29:46.297565 10430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:46.298605 10430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:46.299860 10430 hybrid_clock.cc:648] HybridClock initialized: now 1781922586299795 us; error 60 us; skew 500 ppm
I20260620 02:29:46.300190 10430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:46.302631 10430 webserver.cc:460] Webserver started at http://127.0.97.2:33871/ using document root <none> and password file <none>
I20260620 02:29:46.303345 10430 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:46.303483 10430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:46.303819 10430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:46.306830 10430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "39307b2e3aa54f2f9c084f05111e92c0"
format_stamp: "Formatted at 2026-06-20 02:29:46 on dist-test-slave-r706"
server_key: "63dfa19cee101673813b382d4446db17"
server_key_iv: "aa695e3679773a699048530df151b1ee"
server_key_version: "encryptionkey@0"
I20260620 02:29:46.307629 10430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "39307b2e3aa54f2f9c084f05111e92c0"
format_stamp: "Formatted at 2026-06-20 02:29:46 on dist-test-slave-r706"
server_key: "63dfa19cee101673813b382d4446db17"
server_key_iv: "aa695e3679773a699048530df151b1ee"
server_key_version: "encryptionkey@0"
I20260620 02:29:46.313143 10430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:29:46.316859 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:46.318292 10430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:29:46.318472 10430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "39307b2e3aa54f2f9c084f05111e92c0"
format_stamp: "Formatted at 2026-06-20 02:29:46 on dist-test-slave-r706"
server_key: "63dfa19cee101673813b382d4446db17"
server_key_iv: "aa695e3679773a699048530df151b1ee"
server_key_version: "encryptionkey@0"
I20260620 02:29:46.318715 10430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:46.325443 10430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:46.326335 10430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:46.326637 10430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:46.327651 10430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:46.329263 10430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:46.329396 10430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:46.329535 10430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:46.329627 10430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:46.366767 10430 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46203
I20260620 02:29:46.366825 10556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46203 every 8 connection(s)
I20260620 02:29:46.368310 10430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:46.368848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10430
I20260620 02:29:46.369114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:29:46.369753 388 external_mini_cluster.cc:1428] Setting key 49f58bb6c43a3c59ab1112076e6cf13d
I20260620 02:29:46.374518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34865
--builtin_ntp_servers=127.0.97.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:29:46.377954 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:46.372688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50649 (local address 127.0.97.62:34865)
0620 02:29:46.372877 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:46.372884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:46.373627 (+ 743us) server_negotiation.cc:407] Connection header received
0620 02:29:46.375423 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:46.375432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:46.375519 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:46.375638 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:46.377076 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:46.377084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:46.377091 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:46.377115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:46.377158 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:46.377176 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:46.377303 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:46.377477 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 02:29:46.377548 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:29:46.378829 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:46.372324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34865 (local address 127.0.97.2:50649)
0620 02:29:46.373474 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:46.373510 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:46.375133 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:46.375853 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:46.375861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:46.375953 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:46.376720 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:29:46.376722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:46.376764 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:29:46.376858 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:46.377278 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:46.377286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:46.377291 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:29:46.377512 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 02:29:46.377830 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":469,"threads_started":1}
I20260620 02:29:46.380331 10557 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34865
I20260620 02:29:46.380632 10557 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:46.381302 10557 heartbeater.cc:511] Master 127.0.97.62:34865 requested a full tablet report, sending...
I20260620 02:29:46.382916 10240 ts_manager.cc:194] Registered new tserver with Master: 39307b2e3aa54f2f9c084f05111e92c0 (127.0.97.2:46203)
W20260620 02:29:46.722144 10560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:46.722462 10560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:46.722556 10560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:46.730432 10560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:46.730551 10560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:46.730726 10560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:46.740264 10560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:46.741707 10560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:46.743506 10560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:46.757395 10566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:46.757480 10568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:46.757436 10565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:46.758529 10560 server_base.cc:1034] running on GCE node
I20260620 02:29:46.759207 10560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:46.760233 10560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:46.761488 10560 hybrid_clock.cc:648] HybridClock initialized: now 1781922586761437 us; error 66 us; skew 500 ppm
I20260620 02:29:46.761816 10560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:29:46.764266 10560 webserver.cc:460] Webserver started at http://127.0.97.3:46167/ using document root <none> and password file <none>
I20260620 02:29:46.765053 10560 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:46.765187 10560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:46.765549 10560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:46.768678 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "19cf7fffda2c485e84f9d617e5a41258"
format_stamp: "Formatted at 2026-06-20 02:29:46 on dist-test-slave-r706"
server_key: "89af3c78324217bd8ea674b23db1c639"
server_key_iv: "184d9d95247c7c82dde8f116fa638cdc"
server_key_version: "encryptionkey@0"
I20260620 02:29:46.769665 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "19cf7fffda2c485e84f9d617e5a41258"
format_stamp: "Formatted at 2026-06-20 02:29:46 on dist-test-slave-r706"
server_key: "89af3c78324217bd8ea674b23db1c639"
server_key_iv: "184d9d95247c7c82dde8f116fa638cdc"
server_key_version: "encryptionkey@0"
I20260620 02:29:46.775079 10560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:29:46.778714 10574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:46.780077 10560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:29:46.780256 10560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "19cf7fffda2c485e84f9d617e5a41258"
format_stamp: "Formatted at 2026-06-20 02:29:46 on dist-test-slave-r706"
server_key: "89af3c78324217bd8ea674b23db1c639"
server_key_iv: "184d9d95247c7c82dde8f116fa638cdc"
server_key_version: "encryptionkey@0"
I20260620 02:29:46.780529 10560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:46.786703 10560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:46.787683 10560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:46.787978 10560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:46.788944 10560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:46.790551 10560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:46.790678 10560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:46.790818 10560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:46.790910 10560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:46.824842 10560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38571
I20260620 02:29:46.824900 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38571 every 8 connection(s)
I20260620 02:29:46.826292 10560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:46.836459 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10560
I20260620 02:29:46.836675 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:29:46.837280 388 external_mini_cluster.cc:1428] Setting key a385165218683d97a48c5e98179bec13
I20260620 02:29:46.838294 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:46.828962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33637 (local address 127.0.97.62:34865)
0620 02:29:46.829221 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:46.829228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:46.830639 (+ 1411us) server_negotiation.cc:407] Connection header received
0620 02:29:46.832344 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:46.832348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:46.832406 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:46.832500 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:46.837303 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:46.837308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:46.837311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:46.837365 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:46.837415 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:46.837432 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:46.837585 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:46.837895 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 02:29:46.837971 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 02:29:46.839475 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:46.829478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34865 (local address 127.0.97.3:33637)
0620 02:29:46.830441 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:46.830490 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:46.832062 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:46.832708 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:46.832717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:46.832808 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:46.836806 (+ 3998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:29:46.836810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:46.836876 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:29:46.837050 (+ 174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:46.837606 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:46.837620 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:46.837629 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:29:46.837948 (+ 319us) client_negotiation.cc:240] Negotiation successful
0620 02:29:46.838241 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":308,"threads_started":1}
I20260620 02:29:46.840993 10687 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34865
I20260620 02:29:46.841310 10687 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:46.841845 10687 heartbeater.cc:511] Master 127.0.97.62:34865 requested a full tablet report, sending...
I20260620 02:29:46.843190 10240 ts_manager.cc:194] Registered new tserver with Master: 19cf7fffda2c485e84f9d617e5a41258 (127.0.97.3:38571)
I20260620 02:29:46.853482 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:29:46.859602 10428 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:29:46.858039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.0.97.62:34865)
0620 02:29:46.858260 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:46.858267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:46.858650 (+ 383us) server_negotiation.cc:407] Connection header received
0620 02:29:46.858850 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:46.858857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:46.858925 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:46.859036 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:46.859411 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49110: BlockingRecv error: recv got EOF from 127.0.0.1:49110 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:29:46.860635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10299
I20260620 02:29:46.864177 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:46.861633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.0.97.1:34195)
0620 02:29:46.862151 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:46.862165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:46.862193 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:29:46.862250 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:46.862257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:46.862467 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:46.862600 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:46.863591 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:46.863598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:46.863610 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:46.863623 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:46.863670 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:46.863685 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:46.863799 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:46.863871 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 02:29:46.863956 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260620 02:29:46.865511 10421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:46.946539 10426 heartbeater.cc:503] Master 127.0.97.62:34865 was elected leader, sending a full tablet report...
I20260620 02:29:46.948474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10299
W20260620 02:29:46.967662 10227 connection.cc:535] server connection from 127.0.97.1:55155 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:29:46.968019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10430
I20260620 02:29:46.971477 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:46.968904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.0.97.2:46203)
0620 02:29:46.969547 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:46.969555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:46.969584 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:29:46.969645 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:46.969651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:46.969874 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:46.970011 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:46.970881 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:46.970887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:46.970898 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:46.970914 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:46.970956 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:46.970970 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:46.971079 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:46.971155 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 02:29:46.971234 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":341,"threads_started":1}
I20260620 02:29:46.972703 10552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:47.054306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10430
I20260620 02:29:47.071895 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10560
I20260620 02:29:47.076239 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:47.073062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.0.97.3:38571)
0620 02:29:47.073745 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:47.073752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:47.073783 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:29:47.073845 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:47.073851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:47.074070 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:47.074292 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:47.075468 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:47.075474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:47.075485 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:29:47.075501 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:47.075538 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:29:47.075551 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:47.075747 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:47.075821 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 02:29:47.075924 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":346,"threads_started":1}
I20260620 02:29:47.077641 10682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:47.160640 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10560
I20260620 02:29:47.176563 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10209
I20260620 02:29:47.178004 10270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:47.281618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10209
2026-06-20T02:29:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2472 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:29:47 dist-test-slave-r706 krb5kdc[10705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:29:47 dist-test-slave-r706 krb5kdc[10705](info): set up 2 sockets
Jun 20 02:29:47 dist-test-slave-r706 krb5kdc[10705](info): commencing operation
krb5kdc: starting...
W20260620 02:29:48.394125 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:29:48 dist-test-slave-r706 krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:29:48.660688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38625
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:49.023419 10718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:49.023821 10718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:49.023984 10718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:49.032127 10718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:29:49.032339 10718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:49.032451 10718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:49.032553 10718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:29:49.032646 10718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:29:49.043179 10718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:49.044626 10718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:49.046110 10718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:49.059415 10724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:49.059453 10726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:49.059512 10723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:49.066720 10718 server_base.cc:1034] running on GCE node
I20260620 02:29:49.067648 10718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:49.069316 10718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:49.070897 10718 hybrid_clock.cc:648] HybridClock initialized: now 1781922589070818 us; error 75 us; skew 500 ppm
I20260620 02:29:49.071342 10718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:49 dist-test-slave-r706 krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:49.075012 10718 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:29:49.076757 10718 webserver.cc:460] Webserver started at http://127.0.97.62:34257/ using document root <none> and password file <none>
I20260620 02:29:49.077589 10718 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:49.077723 10718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:49.078034 10718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:49.081197 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "36b786308c3d4bbcbafad32b7f3e359f"
format_stamp: "Formatted at 2026-06-20 02:29:49 on dist-test-slave-r706"
server_key: "1e8ee343e757394d948e4edcf02f8f49"
server_key_iv: "83dde0b351b3fcdf6d1309bff71eb937"
server_key_version: "encryptionkey@0"
I20260620 02:29:49.081997 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "36b786308c3d4bbcbafad32b7f3e359f"
format_stamp: "Formatted at 2026-06-20 02:29:49 on dist-test-slave-r706"
server_key: "1e8ee343e757394d948e4edcf02f8f49"
server_key_iv: "83dde0b351b3fcdf6d1309bff71eb937"
server_key_version: "encryptionkey@0"
I20260620 02:29:49.087587 10718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 02:29:49.091375 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:49.092933 10718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:29:49.093252 10718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "36b786308c3d4bbcbafad32b7f3e359f"
format_stamp: "Formatted at 2026-06-20 02:29:49 on dist-test-slave-r706"
server_key: "1e8ee343e757394d948e4edcf02f8f49"
server_key_iv: "83dde0b351b3fcdf6d1309bff71eb937"
server_key_version: "encryptionkey@0"
I20260620 02:29:49.093535 10718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:49.123407 10718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:49.128222 10718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:49.128572 10718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:49.148253 10718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38625
I20260620 02:29:49.148305 10784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38625 every 8 connection(s)
I20260620 02:29:49.149773 10718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:29:49.153625 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:49.157743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10718
I20260620 02:29:49.158002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:29:49.158850 388 external_mini_cluster.cc:1428] Setting key 34a4c969cd7d1367bea464f6da05a563
I20260620 02:29:49.166618 10785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f: Bootstrap starting.
I20260620 02:29:49.171983 10785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:49.173386 10785 log.cc:824] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:29:49 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:49.177358 10785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f: No bootstrap required, opened a new log
I20260620 02:29:49.185923 10785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b786308c3d4bbcbafad32b7f3e359f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38625 } }
I20260620 02:29:49.186285 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:49.161326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.0.97.62:38625)
0620 02:29:49.162251 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:49.162265 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:49.162311 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:29:49.165177 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:49.165211 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:49.165677 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:49.166063 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:49.168544 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:49.170402 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:49.172372 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:49.173257 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:49.173415 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:49.178437 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:49.178446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:49.178463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:49.178501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:49.181838 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:49.182599 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:49.182616 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:49.182622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:49.182809 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:49.183215 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:49.183220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:49.183230 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:49.183833 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:49.184195 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:49.184658 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 02:29:49.185186 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":440,"threads_started":1}
I20260620 02:29:49.186407 10785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:49.186542 10785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b786308c3d4bbcbafad32b7f3e359f, State: Initialized, Role: FOLLOWER
I20260620 02:29:49.187253 10785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b786308c3d4bbcbafad32b7f3e359f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38625 } }
I20260620 02:29:49.187546 10785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:29:49.187697 10785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:29:49.187916 10785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:49.190610 10785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b786308c3d4bbcbafad32b7f3e359f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38625 } }
I20260620 02:29:49.191151 10785 leader_election.cc:302] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36b786308c3d4bbcbafad32b7f3e359f; no voters:
I20260620 02:29:49.191833 10785 leader_election.cc:288] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:29:49.192010 10790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:49.192744 10790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [term 1 LEADER]: Becoming Leader. State: Replica: 36b786308c3d4bbcbafad32b7f3e359f, State: Running, Role: LEADER
I20260620 02:29:49.193343 10790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b786308c3d4bbcbafad32b7f3e359f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38625 } }
I20260620 02:29:49.194460 10785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:29:49.197976 10792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b786308c3d4bbcbafad32b7f3e359f. Latest consensus state: current_term: 1 leader_uuid: "36b786308c3d4bbcbafad32b7f3e359f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b786308c3d4bbcbafad32b7f3e359f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38625 } } }
I20260620 02:29:49.198706 10792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:49.196820 10791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36b786308c3d4bbcbafad32b7f3e359f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b786308c3d4bbcbafad32b7f3e359f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38625 } } }
I20260620 02:29:49.199076 10791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f [sys.catalog]: This master's current role is: LEADER
I20260620 02:29:49.201112 10799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:29:49.206343 10799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:29:49.216787 10799 catalog_manager.cc:1347] Generated new cluster ID: ed4f4ae21c17483da1504a4608b983d5
I20260620 02:29:49.217128 10799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:29:49.246894 10799 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:29:49.248940 10799 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:29:49.258718 10799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f: Generated new TSK 0
I20260620 02:29:49.259703 10799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:29:49.333884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38625
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:49.685216 10811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:49.685684 10811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:49.685906 10811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:49.693997 10811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:49.694149 10811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:49.694283 10811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:29:49.705150 10811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:49.706548 10811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:49.708029 10811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:49.721767 10816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:49.721879 10819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:49.721740 10817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:49.722765 10811 server_base.cc:1034] running on GCE node
I20260620 02:29:49.723433 10811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:49.724516 10811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:49.725781 10811 hybrid_clock.cc:648] HybridClock initialized: now 1781922589725738 us; error 38 us; skew 500 ppm
I20260620 02:29:49.726135 10811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:49 dist-test-slave-r706 krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:49.729481 10811 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:29:49.731055 10811 webserver.cc:460] Webserver started at http://127.0.97.1:45859/ using document root <none> and password file <none>
I20260620 02:29:49.731850 10811 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:49.732009 10811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:49.732357 10811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:49.735630 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "f37764cdd20f43369778e1315aa52922"
format_stamp: "Formatted at 2026-06-20 02:29:49 on dist-test-slave-r706"
server_key: "0552b8f3db9c67306e7c98f3588a1fe0"
server_key_iv: "a3ca4c36c61709fffff1e005389bebc7"
server_key_version: "encryptionkey@0"
I20260620 02:29:49.736511 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f37764cdd20f43369778e1315aa52922"
format_stamp: "Formatted at 2026-06-20 02:29:49 on dist-test-slave-r706"
server_key: "0552b8f3db9c67306e7c98f3588a1fe0"
server_key_iv: "a3ca4c36c61709fffff1e005389bebc7"
server_key_version: "encryptionkey@0"
I20260620 02:29:49.742508 10811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:29:49.746291 10826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:49.748085 10811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:29:49.748296 10811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "f37764cdd20f43369778e1315aa52922"
format_stamp: "Formatted at 2026-06-20 02:29:49 on dist-test-slave-r706"
server_key: "0552b8f3db9c67306e7c98f3588a1fe0"
server_key_iv: "a3ca4c36c61709fffff1e005389bebc7"
server_key_version: "encryptionkey@0"
I20260620 02:29:49.748528 10811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:49.775354 10811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:49.779428 10811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:49.779734 10811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:49.780699 10811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:49.782089 10811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:49.782194 10811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:49.782351 10811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:49.782433 10811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:49.817689 10811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45493
I20260620 02:29:49.817726 10938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45493 every 8 connection(s)
I20260620 02:29:49.819125 10811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:29:49.827456 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10811
I20260620 02:29:49.827708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:29:49.828511 388 external_mini_cluster.cc:1428] Setting key 2f7892d9f1b64d1a4456b2d972a035ca
Jun 20 02:29:49 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:49.843098 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:49.821810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38429 (local address 127.0.97.62:38625)
0620 02:29:49.822285 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:49.822289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:49.823358 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 02:29:49.825365 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:49.825370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:49.825466 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:49.825570 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:49.827154 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:49.828348 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:49.830880 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:49.831602 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:49.831760 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:49.835659 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:49.835669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:49.835676 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:49.835723 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:49.839348 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:49.840177 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:49.840186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:49.840191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:49.840341 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:49.840982 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:49.840987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:49.840989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:49.841385 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:49.841615 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:49.842614 (+ 999us) server_negotiation.cc:299] Negotiation successful
0620 02:29:49.842808 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260620 02:29:49.844689 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:49.822351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38625 (local address 127.0.97.1:38429)
0620 02:29:49.823092 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:49.823137 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:49.825092 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:49.825786 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:49.825794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:49.826337 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:49.826948 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:49.826965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:49.828717 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:49.828721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:49.830698 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:49.830722 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:49.831835 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:49.831853 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:49.832089 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:49.832848 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:49.832873 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:49.835374 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:49.839515 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:49.839523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:49.839542 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:49.839997 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:49.840492 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:49.840499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:49.840504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:49.840803 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:49.841628 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:49.841640 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:49.842142 (+ 502us) client_negotiation.cc:769] Sending connection context
0620 02:29:49.842465 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 02:29:49.843129 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":303,"threads_started":1}
I20260620 02:29:49.846504 10939 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38625
I20260620 02:29:49.846899 10939 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:49.847781 10939 heartbeater.cc:511] Master 127.0.97.62:38625 requested a full tablet report, sending...
I20260620 02:29:49.850103 10750 ts_manager.cc:194] Registered new tserver with Master: f37764cdd20f43369778e1315aa52922 (127.0.97.1:45493)
I20260620 02:29:49.851610 10750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38429
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:29:49.888839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38625
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:50.235502 10945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:50.235873 10945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:50.236059 10945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:50.243690 10945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:50.243863 10945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:50.244057 10945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:29:50.254410 10945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:50.255792 10945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:50.257236 10945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:50.271286 10953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:50.271270 10951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:50.271638 10950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:50.272095 10945 server_base.cc:1034] running on GCE node
I20260620 02:29:50.272742 10945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:50.273816 10945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:50.275043 10945 hybrid_clock.cc:648] HybridClock initialized: now 1781922590274992 us; error 42 us; skew 500 ppm
I20260620 02:29:50.275426 10945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:50 dist-test-slave-r706 krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:50.278838 10945 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:29:50.280442 10945 webserver.cc:460] Webserver started at http://127.0.97.2:38091/ using document root <none> and password file <none>
I20260620 02:29:50.281214 10945 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:50.281323 10945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:50.281582 10945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:50.284571 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb"
format_stamp: "Formatted at 2026-06-20 02:29:50 on dist-test-slave-r706"
server_key: "04165d261915421b49455bec54923fe4"
server_key_iv: "c6c3bed2ce443bcfff338a0d2950dddd"
server_key_version: "encryptionkey@0"
I20260620 02:29:50.285451 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb"
format_stamp: "Formatted at 2026-06-20 02:29:50 on dist-test-slave-r706"
server_key: "04165d261915421b49455bec54923fe4"
server_key_iv: "c6c3bed2ce443bcfff338a0d2950dddd"
server_key_version: "encryptionkey@0"
I20260620 02:29:50.290776 10945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:29:50.294481 10960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:50.296105 10945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:29:50.296245 10945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb"
format_stamp: "Formatted at 2026-06-20 02:29:50 on dist-test-slave-r706"
server_key: "04165d261915421b49455bec54923fe4"
server_key_iv: "c6c3bed2ce443bcfff338a0d2950dddd"
server_key_version: "encryptionkey@0"
I20260620 02:29:50.296420 10945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:50.316900 10945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:50.321235 10945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:50.321485 10945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:50.322449 10945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:50.323787 10945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:50.323868 10945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:50.323951 10945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:50.323997 10945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:50.360528 10945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45755
I20260620 02:29:50.360607 11072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45755 every 8 connection(s)
I20260620 02:29:50.361901 10945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:29:50.366289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 10945
I20260620 02:29:50.366482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:29:50.366900 388 external_mini_cluster.cc:1428] Setting key 2e3c770c333f6831636f71c67eb815ce
Jun 20 02:29:50 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:50.382973 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.364457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43013 (local address 127.0.97.62:38625)
0620 02:29:50.364914 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:50.364919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:50.365987 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 02:29:50.368043 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:50.368047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:50.368113 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:50.368227 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:50.369981 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.370714 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.372308 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.372761 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.372904 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:50.376832 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:50.376838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:50.376842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:50.376877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:50.379682 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:50.380480 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:50.380485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:50.380487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:50.380618 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:50.381121 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:50.381129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:50.381136 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:50.381427 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:50.381597 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:50.382330 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 02:29:50.382572 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":240,"threads_started":1}
I20260620 02:29:50.383998 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.364993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38625 (local address 127.0.97.2:43013)
0620 02:29:50.365792 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:50.365841 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:50.367733 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:50.368480 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:50.368491 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:50.369222 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:50.369795 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:50.369812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.370887 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.370896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:50.372143 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:50.372164 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.373059 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.373068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:50.373309 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:50.374052 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:50.374081 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:50.376534 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:50.379859 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:50.379869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:50.379888 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:50.380332 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:50.380752 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:50.380757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:50.380763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:50.380958 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:50.381566 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:50.381574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:50.381948 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 02:29:50.382223 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 02:29:50.382836 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":295,"threads_started":1}
I20260620 02:29:50.385634 11073 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38625
I20260620 02:29:50.386008 11073 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:50.387014 11073 heartbeater.cc:511] Master 127.0.97.62:38625 requested a full tablet report, sending...
I20260620 02:29:50.389096 10750 ts_manager.cc:194] Registered new tserver with Master: 9f9348d8b3e646f98d1ba8be0fa3c3cb (127.0.97.2:45755)
I20260620 02:29:50.390494 10750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43013
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:29:50.426841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38625
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:29:50.771381 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:29:50.771765 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:29:50.771970 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:29:50.779330 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:29:50.779450 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:29:50.779650 11079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:29:50.789873 11079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:29:50.791266 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:29:50.792685 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:29:50.807020 11085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:50.807089 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:29:50.807106 11084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:29:50.807911 11079 server_base.cc:1034] running on GCE node
I20260620 02:29:50.808532 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:29:50.809579 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:29:50.810830 11079 hybrid_clock.cc:648] HybridClock initialized: now 1781922590810780 us; error 42 us; skew 500 ppm
I20260620 02:29:50.811188 11079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:29:50 dist-test-slave-r706 krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:29:50.814404 11079 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:29:50.815955 11079 webserver.cc:460] Webserver started at http://127.0.97.3:37279/ using document root <none> and password file <none>
I20260620 02:29:50.816666 11079 fs_manager.cc:359] Metadata directory not provided
I20260620 02:29:50.816812 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:29:50.817222 11079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:29:50.820114 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "7dae0e3419874274b4cf8c6e6c268deb"
format_stamp: "Formatted at 2026-06-20 02:29:50 on dist-test-slave-r706"
server_key: "2364115f01f0642518681c5523ec2ee2"
server_key_iv: "0c67b8a5c667b8b98f0c1ff7c4b988dd"
server_key_version: "encryptionkey@0"
I20260620 02:29:50.820889 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7dae0e3419874274b4cf8c6e6c268deb"
format_stamp: "Formatted at 2026-06-20 02:29:50 on dist-test-slave-r706"
server_key: "2364115f01f0642518681c5523ec2ee2"
server_key_iv: "0c67b8a5c667b8b98f0c1ff7c4b988dd"
server_key_version: "encryptionkey@0"
I20260620 02:29:50.826273 11079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:29:50.829778 11094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:50.831140 11079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:29:50.831313 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "7dae0e3419874274b4cf8c6e6c268deb"
format_stamp: "Formatted at 2026-06-20 02:29:50 on dist-test-slave-r706"
server_key: "2364115f01f0642518681c5523ec2ee2"
server_key_iv: "0c67b8a5c667b8b98f0c1ff7c4b988dd"
server_key_version: "encryptionkey@0"
I20260620 02:29:50.831513 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:29:50.855041 10939 heartbeater.cc:503] Master 127.0.97.62:38625 was elected leader, sending a full tablet report...
I20260620 02:29:50.864231 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:29:50.868705 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:29:50.869077 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:29:50.870131 11079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:29:50.871537 11079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:29:50.871646 11079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:50.871810 11079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:29:50.871884 11079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:29:50.906772 11079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40419
I20260620 02:29:50.906819 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40419 every 8 connection(s)
I20260620 02:29:50.908138 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:29:50.914106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 11079
I20260620 02:29:50.914345 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:29:50.914935 388 external_mini_cluster.cc:1428] Setting key 094e3b752bda4e0f3242367f09c604c8
Jun 20 02:29:50 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:29:50.930248 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.910851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50915 (local address 127.0.97.62:38625)
0620 02:29:50.911371 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:50.911376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:50.912316 (+ 940us) server_negotiation.cc:407] Connection header received
0620 02:29:50.914554 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:50.914564 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:50.914638 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:50.914736 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:50.916715 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.917694 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.919536 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.919973 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.920080 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:50.923932 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:50.923940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:50.923943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:50.923978 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:50.926794 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:50.927639 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:50.927643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:50.927645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:50.927768 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:50.928195 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:50.928200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:50.928202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:50.928504 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:50.928655 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:50.929588 (+ 933us) server_negotiation.cc:299] Negotiation successful
0620 02:29:50.929867 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":265,"threads_started":1}
I20260620 02:29:50.931411 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.911439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38625 (local address 127.0.97.3:50915)
0620 02:29:50.912144 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:50.912182 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:50.914204 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:50.915055 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:50.915067 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:50.915757 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:29:50.916527 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:50.916557 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.917855 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.917861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:50.919357 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:50.919371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.920237 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.920244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:50.920501 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:50.921306 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:29:50.921339 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:29:50.923710 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:29:50.926962 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:50.926970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:50.926984 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:50.927482 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:50.927866 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:29:50.927869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:29:50.927872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:29:50.928074 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:29:50.928695 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:29:50.928703 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:29:50.929200 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 02:29:50.929537 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 02:29:50.930056 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":276,"threads_started":1}
I20260620 02:29:50.932963 11207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38625
I20260620 02:29:50.933341 11207 heartbeater.cc:464] Registering TS with master...
I20260620 02:29:50.934082 11207 heartbeater.cc:511] Master 127.0.97.62:38625 requested a full tablet report, sending...
I20260620 02:29:50.935680 10750 ts_manager.cc:194] Registered new tserver with Master: 7dae0e3419874274b4cf8c6e6c268deb (127.0.97.3:40419)
I20260620 02:29:50.936519 10750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50915
I20260620 02:29:50.943437 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:29:50.966004 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.948218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.97.62:38625)
0620 02:29:50.948401 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:50.948406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:50.948970 (+ 564us) server_negotiation.cc:407] Connection header received
0620 02:29:50.949204 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:50.949209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:50.949276 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:50.949363 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:50.950848 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.951609 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.953555 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:50.954377 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:50.954542 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:50.958210 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:50.958231 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:50.958239 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:50.958284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:50.962112 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:50.962987 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:50.962993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:50.962996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:50.963150 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:50.963607 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:50.963612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:50.963615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:50.964035 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:50.964261 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:50.965443 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0620 02:29:50.965675 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:29:50.971068 10750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:29:50.978212 10750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:29:51.019338 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.991563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.97.1:45493)
0620 02:29:51.005693 (+ 14130us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.005706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.011088 (+ 5382us) server_negotiation.cc:407] Connection header received
0620 02:29:51.011567 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.011579 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.011850 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.012094 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:51.012677 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.014228 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.016533 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.018022 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.018172 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.018417 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.018804 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.018975 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13927,"thread_start_us":13741,"threads_started":1}
I20260620 02:29:51.019687 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.990903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45755 (local address 127.0.0.1:58360)
0620 02:29:51.003702 (+ 12799us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:51.003766 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:51.003969 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:51.010879 (+ 6910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:51.010890 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:51.010952 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:51.011609 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.011624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.013654 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.013659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.016753 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.016767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.018896 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.018904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.019053 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.019081 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:29:51.019169 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:29:51.019304 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12460,"mutex_wait_us":79,"thread_start_us":12293,"threads_started":1}
I20260620 02:29:51.020133 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.991047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45493 (local address 127.0.0.1:59296)
0620 02:29:51.010829 (+ 19782us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:51.010851 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:51.011272 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:51.012059 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:51.012064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:51.012092 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:51.012495 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.012509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.014399 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.014408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.016373 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.016385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.018253 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.018263 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.018468 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.018534 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 02:29:51.018738 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 02:29:51.018966 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19636,"mutex_wait_us":73,"spinlock_wait_cycles":1280,"thread_start_us":20029,"threads_started":1}
I20260620 02:29:51.020330 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.992412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.0.97.3:40419)
0620 02:29:51.010322 (+ 17910us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.010330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.011587 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 02:29:51.011807 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.011815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.012031 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.012231 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:51.013138 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.014652 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.017226 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.018197 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.018323 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.018543 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.019779 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.019971 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17742,"thread_start_us":17601,"threads_started":1}
I20260620 02:29:51.020612 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.991647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40419 (local address 127.0.0.1:56102)
0620 02:29:51.011441 (+ 19794us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:51.011464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:51.011625 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:51.012497 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:51.012502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:51.012526 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:51.012942 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.012953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.014803 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.014810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.017070 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.017089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.018591 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.018598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.018778 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.018820 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:29:51.018908 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:29:51.019052 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19648,"mutex_wait_us":27,"thread_start_us":18846,"threads_started":1}
I20260620 02:29:51.020975 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:50.991073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.97.2:45755)
0620 02:29:51.009877 (+ 18804us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.009888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.009928 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:29:51.010045 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.010057 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.010394 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.010641 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:51.011833 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.013483 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.016933 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.018408 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.018563 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.018745 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.020401 (+ 1656us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.020609 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18633,"thread_start_us":18523,"threads_started":1}
I20260620 02:29:51.024125 11142 tablet_service.cc:1467] Processing CreateTablet for tablet bfc462e3264746ddb1b0701ab8894456 (DEFAULT_TABLE table=test-table [id=c4792456245b488c91aeaa782fafec5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:51.024214 11008 tablet_service.cc:1467] Processing CreateTablet for tablet bfc462e3264746ddb1b0701ab8894456 (DEFAULT_TABLE table=test-table [id=c4792456245b488c91aeaa782fafec5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:51.024212 10874 tablet_service.cc:1467] Processing CreateTablet for tablet bfc462e3264746ddb1b0701ab8894456 (DEFAULT_TABLE table=test-table [id=c4792456245b488c91aeaa782fafec5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:29:51.025578 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc462e3264746ddb1b0701ab8894456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:51.025614 10874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc462e3264746ddb1b0701ab8894456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:51.025645 11008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc462e3264746ddb1b0701ab8894456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:29:51.037690 11226 tablet_bootstrap.cc:492] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Bootstrap starting.
I20260620 02:29:51.040946 11226 tablet_bootstrap.cc:654] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:51.041543 11228 tablet_bootstrap.cc:492] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Bootstrap starting.
I20260620 02:29:51.042560 11226 log.cc:824] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:51.044227 11227 tablet_bootstrap.cc:492] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Bootstrap starting.
I20260620 02:29:51.045593 11228 tablet_bootstrap.cc:654] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:51.046938 11226 tablet_bootstrap.cc:492] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: No bootstrap required, opened a new log
I20260620 02:29:51.046931 11228 log.cc:824] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:51.047235 11226 ts_tablet_manager.cc:1397] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 02:29:51.050340 11228 tablet_bootstrap.cc:492] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: No bootstrap required, opened a new log
I20260620 02:29:51.050640 11228 ts_tablet_manager.cc:1397] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:29:51.051429 11227 tablet_bootstrap.cc:654] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Neither blocks nor log segments found. Creating new log.
I20260620 02:29:51.052794 11227 log.cc:824] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Log is configured to *not* fsync() on all Append() calls
I20260620 02:29:51.056145 11227 tablet_bootstrap.cc:492] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: No bootstrap required, opened a new log
I20260620 02:29:51.056478 11227 ts_tablet_manager.cc:1397] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260620 02:29:51.058287 11226 raft_consensus.cc:348] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.058769 11226 raft_consensus.cc:374] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:51.058877 11226 raft_consensus.cc:729] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dae0e3419874274b4cf8c6e6c268deb, State: Initialized, Role: FOLLOWER
I20260620 02:29:51.058734 11228 raft_consensus.cc:348] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.059221 11228 raft_consensus.cc:374] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:51.059342 11228 raft_consensus.cc:729] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9348d8b3e646f98d1ba8be0fa3c3cb, State: Initialized, Role: FOLLOWER
I20260620 02:29:51.059672 11226 consensus_queue.cc:260] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.059952 11228 consensus_queue.cc:260] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.062958 11207 heartbeater.cc:503] Master 127.0.97.62:38625 was elected leader, sending a full tablet report...
I20260620 02:29:51.065099 11226 ts_tablet_manager.cc:1428] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Time spent starting tablet: real 0.018s user 0.008s sys 0.010s
I20260620 02:29:51.066072 11073 heartbeater.cc:503] Master 127.0.97.62:38625 was elected leader, sending a full tablet report...
I20260620 02:29:51.067180 11228 ts_tablet_manager.cc:1428] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260620 02:29:51.068171 11227 raft_consensus.cc:348] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.068614 11227 raft_consensus.cc:374] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:29:51.068734 11227 raft_consensus.cc:729] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37764cdd20f43369778e1315aa52922, State: Initialized, Role: FOLLOWER
I20260620 02:29:51.069401 11227 consensus_queue.cc:260] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.071463 11227 ts_tablet_manager.cc:1428] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260620 02:29:51.071724 10940 tablet.cc:2369] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:29:51.114060 11074 tablet.cc:2369] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:29:51.159901 11208 tablet.cc:2369] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:29:51.214118 11232 raft_consensus.cc:482] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:29:51.214479 11232 raft_consensus.cc:504] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.216236 11232 leader_election.cc:288] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f9348d8b3e646f98d1ba8be0fa3c3cb (127.0.97.2:45755), f37764cdd20f43369778e1315aa52922 (127.0.97.1:45493)
I20260620 02:29:51.222576 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.216804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36097 (local address 127.0.97.2:45755)
0620 02:29:51.217056 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.217063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.217088 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:29:51.217173 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.217180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.217268 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.217378 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:51.217846 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.218816 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.220664 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.221686 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.221797 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.221869 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.222247 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.222335 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:29:51.222687 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.216513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45755 (local address 127.0.97.3:36097)
0620 02:29:51.216808 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:51.216833 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:51.217024 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:51.217431 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:51.217435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:51.217460 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:51.217689 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.217702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.218959 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.218965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.220493 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.220506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.221877 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.221884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.222077 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.222117 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:29:51.222226 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 02:29:51.222364 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 02:29:51.223380 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.216717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60587 (local address 127.0.97.1:45493)
0620 02:29:51.216927 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.216933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.217587 (+ 654us) server_negotiation.cc:407] Connection header received
0620 02:29:51.217708 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.217713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.217801 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.217890 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:29:51.218484 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.219386 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.221312 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.222380 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.222509 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.222606 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.223018 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.223150 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:29:51.223920 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.216531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45493 (local address 127.0.97.3:60587)
0620 02:29:51.217460 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 02:29:51.217477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:29:51.217599 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:29:51.217930 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:29:51.217937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:29:51.217964 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:29:51.218329 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.218341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.219545 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.219548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.221149 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:29:51.221163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.222594 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.222603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:29:51.222791 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.222824 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:29:51.223565 (+ 741us) client_negotiation.cc:240] Negotiation successful
0620 02:29:51.223665 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":159,"thread_start_us":408,"threads_started":1}
I20260620 02:29:51.223779 11028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc462e3264746ddb1b0701ab8894456" candidate_uuid: "7dae0e3419874274b4cf8c6e6c268deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" is_pre_election: true
I20260620 02:29:51.224332 11028 raft_consensus.cc:2454] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dae0e3419874274b4cf8c6e6c268deb in term 0.
I20260620 02:29:51.224906 10894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc462e3264746ddb1b0701ab8894456" candidate_uuid: "7dae0e3419874274b4cf8c6e6c268deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37764cdd20f43369778e1315aa52922" is_pre_election: true
I20260620 02:29:51.225185 11097 leader_election.cc:302] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dae0e3419874274b4cf8c6e6c268deb, 9f9348d8b3e646f98d1ba8be0fa3c3cb; no voters:
I20260620 02:29:51.225546 10894 raft_consensus.cc:2454] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dae0e3419874274b4cf8c6e6c268deb in term 0.
I20260620 02:29:51.225602 11232 raft_consensus.cc:2781] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:29:51.225765 11232 raft_consensus.cc:482] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:29:51.225895 11232 raft_consensus.cc:3037] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:51.228260 11232 raft_consensus.cc:504] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.228847 11232 leader_election.cc:288] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [CANDIDATE]: Term 1 election: Requested vote from peers 9f9348d8b3e646f98d1ba8be0fa3c3cb (127.0.97.2:45755), f37764cdd20f43369778e1315aa52922 (127.0.97.1:45493)
I20260620 02:29:51.229343 11028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc462e3264746ddb1b0701ab8894456" candidate_uuid: "7dae0e3419874274b4cf8c6e6c268deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb"
I20260620 02:29:51.229375 10894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc462e3264746ddb1b0701ab8894456" candidate_uuid: "7dae0e3419874274b4cf8c6e6c268deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37764cdd20f43369778e1315aa52922"
I20260620 02:29:51.229573 10894 raft_consensus.cc:3037] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:51.229573 11028 raft_consensus.cc:3037] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:29:51.232834 11028 raft_consensus.cc:2454] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dae0e3419874274b4cf8c6e6c268deb in term 1.
I20260620 02:29:51.232834 10894 raft_consensus.cc:2454] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dae0e3419874274b4cf8c6e6c268deb in term 1.
I20260620 02:29:51.233319 11097 leader_election.cc:302] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dae0e3419874274b4cf8c6e6c268deb, 9f9348d8b3e646f98d1ba8be0fa3c3cb; no voters:
I20260620 02:29:51.233654 11232 raft_consensus.cc:2781] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:29:51.234287 11232 raft_consensus.cc:686] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 1 LEADER]: Becoming Leader. State: Replica: 7dae0e3419874274b4cf8c6e6c268deb, State: Running, Role: LEADER
I20260620 02:29:51.234849 11232 consensus_queue.cc:237] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } }
I20260620 02:29:51.239427 10750 catalog_manager.cc:5495] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7dae0e3419874274b4cf8c6e6c268deb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7dae0e3419874274b4cf8c6e6c268deb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dae0e3419874274b4cf8c6e6c268deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40419 } health_report { overall_health: HEALTHY } } }
I20260620 02:29:51.306216 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.300767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.0.97.3:40419)
0620 02:29:51.300989 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.301035 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.301061 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:29:51.301255 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.301259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.301387 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.301537 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:29:51.301910 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.302641 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.303970 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.304376 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.304470 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.304996 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:29:51.305576 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:29:51.305694 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.305818 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.305891 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:29:51.315110 11028 raft_consensus.cc:1261] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 1 FOLLOWER]: Refusing update from remote peer 7dae0e3419874274b4cf8c6e6c268deb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:51.315110 10894 raft_consensus.cc:1261] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 1 FOLLOWER]: Refusing update from remote peer 7dae0e3419874274b4cf8c6e6c268deb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:29:51.316308 11236 consensus_queue.cc:1035] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9348d8b3e646f98d1ba8be0fa3c3cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:51.316705 11232 consensus_queue.cc:1035] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37764cdd20f43369778e1315aa52922" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:29:51.330986 11241 mvcc.cc:205] Tried to move back new op lower bound from 7298754934013370368 to 7298754933701799936. Current Snapshot: MvccSnapshot[applied={T|T < 7298754934013370368}]
I20260620 02:29:51.345521 10748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:51.345839 10748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:29:51.349924 10748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36b786308c3d4bbcbafad32b7f3e359f: Sending DeleteTablet for 3 replicas of tablet bfc462e3264746ddb1b0701ab8894456
I20260620 02:29:51.351155 11008 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc462e3264746ddb1b0701ab8894456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58360
I20260620 02:29:51.351250 10874 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc462e3264746ddb1b0701ab8894456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59296
I20260620 02:29:51.351253 11142 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc462e3264746ddb1b0701ab8894456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:29:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56102
I20260620 02:29:51.352874 11248 tablet_replica.cc:330] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: stopping tablet replica
I20260620 02:29:51.353340 11248 raft_consensus.cc:2229] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:51.353807 11248 raft_consensus.cc:2258] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:51.352962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10811
I20260620 02:29:51.356313 11248 ts_tablet_manager.cc:1905] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:51.366600 11248 ts_tablet_manager.cc:1918] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:51.366798 11248 log.cc:1196] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/bfc462e3264746ddb1b0701ab8894456
I20260620 02:29:51.366991 11250 tablet_replica.cc:330] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: stopping tablet replica
I20260620 02:29:51.367462 11248 ts_tablet_manager.cc:1939] T bfc462e3264746ddb1b0701ab8894456 P 9f9348d8b3e646f98d1ba8be0fa3c3cb: Deleting consensus metadata
I20260620 02:29:51.367484 11250 raft_consensus.cc:2229] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:29:51.368294 11250 raft_consensus.cc:2258] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:51.369113 11249 tablet_replica.cc:330] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: stopping tablet replica
I20260620 02:29:51.369534 11249 raft_consensus.cc:2229] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:29:51.369755 10736 catalog_manager.cc:4841] TS 9f9348d8b3e646f98d1ba8be0fa3c3cb (127.0.97.2:45755): tablet bfc462e3264746ddb1b0701ab8894456 (table test-table [id=c4792456245b488c91aeaa782fafec5a]) successfully deleted
I20260620 02:29:51.370088 11249 raft_consensus.cc:2258] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:29:51.371868 11250 ts_tablet_manager.cc:1905] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:51.372259 11249 ts_tablet_manager.cc:1905] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:29:51.379295 11250 ts_tablet_manager.cc:1918] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:51.379295 11249 ts_tablet_manager.cc:1918] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:29:51.379477 11250 log.cc:1196] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/bfc462e3264746ddb1b0701ab8894456
I20260620 02:29:51.379474 11249 log.cc:1196] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/bfc462e3264746ddb1b0701ab8894456
I20260620 02:29:51.380044 11249 ts_tablet_manager.cc:1939] T bfc462e3264746ddb1b0701ab8894456 P f37764cdd20f43369778e1315aa52922: Deleting consensus metadata
I20260620 02:29:51.380044 11250 ts_tablet_manager.cc:1939] T bfc462e3264746ddb1b0701ab8894456 P 7dae0e3419874274b4cf8c6e6c268deb: Deleting consensus metadata
I20260620 02:29:51.381963 10735 catalog_manager.cc:4841] TS 7dae0e3419874274b4cf8c6e6c268deb (127.0.97.3:40419): tablet bfc462e3264746ddb1b0701ab8894456 (table test-table [id=c4792456245b488c91aeaa782fafec5a]) successfully deleted
I20260620 02:29:51.382382 10734 catalog_manager.cc:4841] TS f37764cdd20f43369778e1315aa52922 (127.0.97.1:45493): tablet bfc462e3264746ddb1b0701ab8894456 (table test-table [id=c4792456245b488c91aeaa782fafec5a]) successfully deleted
Jun 20 02:29:51 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:29:51.400347 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.356504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.0.97.1:45493)
0620 02:29:51.356754 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.356764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.381492 (+ 24728us) server_negotiation.cc:407] Connection header received
0620 02:29:51.382385 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.382392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.382482 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.382602 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:51.383772 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.384866 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.386771 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.387467 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.387620 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.392883 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:51.392901 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:51.392918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:51.392963 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:51.396836 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:51.397916 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:51.397924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:51.397927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:51.398103 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:51.398560 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:51.398564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:51.398566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:51.399042 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:51.399321 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.399835 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.400012 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:29:51.401544 10934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:51.492954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10811
I20260620 02:29:51.518827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10945
Jun 20 02:29:51 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:29:51.540776 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.519709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.97.2:45755)
0620 02:29:51.519935 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.519941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.519962 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:29:51.520068 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.520073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.520145 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.520261 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:51.521664 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.522410 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.523739 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.524371 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.524508 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.531435 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:51.531502 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:51.531528 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:51.531619 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:51.537351 (+ 5732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:51.538455 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:51.538465 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:51.538468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:51.538664 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:51.539121 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:51.539127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:51.539130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:51.539549 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:51.539851 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.540173 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.540418 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:29:51.541636 11068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:51.646178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10945
I20260620 02:29:51.670339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 11079
Jun 20 02:29:51 dist-test-slave-r706 krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:29:51.689349 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:29:51.671195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.0.97.3:40419)
0620 02:29:51.671371 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:29:51.671376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:29:51.671400 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:29:51.671649 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:29:51.671654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:29:51.671730 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:29:51.671838 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:29:51.673605 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.674459 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.676245 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:29:51.676702 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:29:51.676808 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:29:51.682623 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:29:51.682634 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:29:51.682649 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:29:51.682689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:29:51.686097 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:51.686989 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:51.686996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:51.686998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:51.687138 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:29:51.687650 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:29:51.687654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:29:51.687656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:29:51.688027 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:29:51.688260 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:29:51.688645 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 02:29:51.688814 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:29:51.690068 11202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:51.809834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 11079
I20260620 02:29:51.829290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 10718
I20260620 02:29:51.830478 10780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:29:51.940260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 10718
2026-06-20T02:29:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4659 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:29:52 dist-test-slave-r706 krb5kdc[11258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:29:52 dist-test-slave-r706 krb5kdc[11258](info): set up 2 sockets
Jun 20 02:29:52 dist-test-slave-r706 krb5kdc[11258](info): commencing operation
krb5kdc: starting...
W20260620 02:29:53.085947 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:29:53 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:29:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 02:29:53.454034 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkyewzx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/postgres -l logfile start
2026-06-20 02:29:56.158 UTC [11288] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:29:56.158 UTC [11288] LOG: listening on IPv4 address "127.0.97.20", port 41395
2026-06-20 02:29:56.161 UTC [11288] LOG: listening on Unix socket "/tmp/.s.PGSQL.41395"
2026-06-20 02:29:56.185 UTC [11291] LOG: database system was shut down at 2026-06-20 02:29:54 UTC
2026-06-20 02:29:56.189 UTC [11288] LOG: database system is ready to accept connections
I20260620 02:29:57.189594 388 mini_postgres.cc:96] Postgres bound to 41395
2026-06-20 02:29:57.210 UTC [11299] FATAL: database "slave" does not exist
127.0.97.20:41395 - accepting connections
I20260620 02:29:57.211100 388 mini_ranger.cc:161] Starting Ranger...
I20260620 02:29:57.240899 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:29:57.770635 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:29:57.770754 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-admin
2026-06-20 02:29:58,073 [I] DB FLAVOR :POSTGRES
2026-06-20 02:29:58,074 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:29:58,074 [I] Checking connection..
2026-06-20 02:29:58,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:29:58,336 [I] Checking connection passed.
2026-06-20 02:29:58,337 [I] --------- Verifying version history table ---------
2026-06-20 02:29:58,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:29:58,586 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:29:58,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:29:58,857 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:29:58,858 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:29:58,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:29:59.110 UTC [11363] WARNING: there is no transaction in progress
2026-06-20 02:29:59,116 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:29:59,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:29:59,390 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:29:59,391 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:29:59,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:29:59,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:29:59,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r706.c.gcp-upstream.internal','N') ;"
2026-06-20 02:30:00,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:30:00,451 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:30:00,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:30:00,729 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:30:00,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:30:01,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:30:01,263 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:30:01,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:30:02.387 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.406 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.425 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.444 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.461 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.483 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.545 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.553 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.569 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.582 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.594 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.610 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.621 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.629 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.636 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:02.644 UTC [11471] WARNING: there is no transaction in progress
2026-06-20 02:30:03,392 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:30:03,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:30:03,666 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:30:03,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:30:03,954 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:30:03,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:30:04,237 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:30:04,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:30:04,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:30:04,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r706.c.gcp-upstream.internal';"
2026-06-20 02:30:05,080 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:30:05,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:30:05,332 [I] DB_PATCHES have already been applied
I20260620 02:30:05.338500 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkyewzx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:30:05.338603 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 02:30:05.356663 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:40197
Jun 20, 2026 2:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:30:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:30:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:30:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:30:06.511333 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.300s user 0.005s sys 0.046s
Jun 20, 2026 2:30:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:30:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:30:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:30:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:30:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:30:29.309345 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:30:29 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922629, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:30:29.524585 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:30:29.552222 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:30:30.553535 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:30:30.553710 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms
I20260620 02:30:30.553848 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:41395
2026-06-20 02:30:31,057 [I] DB FLAVOR :POSTGRES
2026-06-20 02:30:31,057 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:30:31,057 [I] Checking connection
2026-06-20 02:30:31,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:30:31,292 [I] connection success
2026-06-20 02:30:31,292 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:30:31,292 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:30:31,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:30:31,532 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:30:31,533 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:30:31,533 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:30:31,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:30:31,793 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:30:32.333724 388 mini_ranger_kms.cc:310] Created kms service
I20260620 02:30:32.504958 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:30:32.549849 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:30:32 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922629, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:30:32.959373 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:30:32.959698 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:30:32.959757 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 02:30:32.980681 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11652
I20260620 02:30:32.980935 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:45225
Jun 20, 2026 2:30:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:30:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:30:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:30:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:30:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:30:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 02:30:35 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922635, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:30:35.345525 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.821s user 0.125s sys 0.414s
I20260620 02:30:35.345778 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:30:35.345980 388 mini_ranger_kms.cc:401] 127.0.97.20:45225/kms/v1/keys
Jun 20 02:30:35 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922635, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:30:36 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922635, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:30:38 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:30:38.452654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33065
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33065
--ranger_config_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45225/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:30:38.961395 11702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:30:38.961884 11702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:30:38.962124 11702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:30:38.971135 11702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:30:38.971343 11702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:30:38.971467 11702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:30:38.971609 11702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:30:38.971732 11702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:30:38.984026 11702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:45225/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33065
--ranger_config_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:30:38.985554 11702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:30:38.986963 11702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:30:39.002421 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:39.002616 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:39.002627 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:30:39.003677 11702 server_base.cc:1034] running on GCE node
I20260620 02:30:39.004467 11702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:30:39.006088 11702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:30:39.007359 11702 hybrid_clock.cc:648] HybridClock initialized: now 1781922639007268 us; error 52 us; skew 500 ppm
I20260620 02:30:39.007779 11702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:30:39 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:30:39.017812 11702 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:30:39.020084 11702 webserver.cc:460] Webserver started at http://127.0.97.62:44699/ using document root <none> and password file <none>
I20260620 02:30:39.020957 11702 fs_manager.cc:359] Metadata directory not provided
I20260620 02:30:39.021191 11702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:30:39.021543 11702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:30:39 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:30:39.081884 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "129255ec40944dd098734ec1e0994029"
format_stamp: "Formatted at 2026-06-20 02:30:39 on dist-test-slave-r706"
server_key: "4a603b342d6d96ad7f323ce64ff1f5a0"
server_key_iv: "74656c9f670715bcd63df003900ce224"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:39.082891 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "129255ec40944dd098734ec1e0994029"
format_stamp: "Formatted at 2026-06-20 02:30:39 on dist-test-slave-r706"
server_key: "4a603b342d6d96ad7f323ce64ff1f5a0"
server_key_iv: "74656c9f670715bcd63df003900ce224"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:39.089910 11702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:30:39.108109 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:39.109784 11702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:30:39.110020 11702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "129255ec40944dd098734ec1e0994029"
format_stamp: "Formatted at 2026-06-20 02:30:39 on dist-test-slave-r706"
server_key: "4a603b342d6d96ad7f323ce64ff1f5a0"
server_key_iv: "74656c9f670715bcd63df003900ce224"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:39.110373 11702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:30:39.129739 11702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:30:39.134775 11702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:30:39.135185 11702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:30:39.171432 11702 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33065
I20260620 02:30:39.171537 11768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33065 every 8 connection(s)
I20260620 02:30:39.173022 11702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:30:39.177525 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:30:39.179494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 11702
I20260620 02:30:39.179723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:30:39.180401 388 external_mini_cluster.cc:1428] Setting key 604a111e0747bc87551816cc65dbdf8a
I20260620 02:30:39.191408 11769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029: Bootstrap starting.
I20260620 02:30:39.195879 11769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029: Neither blocks nor log segments found. Creating new log.
Jun 20 02:30:39 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:30:39.197507 11769 log.cc:824] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029: Log is configured to *not* fsync() on all Append() calls
I20260620 02:30:39.201223 11769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029: No bootstrap required, opened a new log
I20260620 02:30:39.205945 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:39.182757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.0.97.62:33065)
0620 02:30:39.184036 (+ 1279us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:39.184050 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:39.184104 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:30:39.186480 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:39.186504 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:39.186894 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:39.187308 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:39.188699 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:39.190447 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:39.191765 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:39.192575 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:39.192767 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:39.197523 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:39.197532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:39.197546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:39.197619 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:39.201424 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:39.202107 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:39.202115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:39.202121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:39.202296 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:39.202721 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:39.202731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:39.202734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:39.203357 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:39.203779 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:39.204224 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 02:30:39.204664 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":691,"threads_started":1}
I20260620 02:30:39.212152 11769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129255ec40944dd098734ec1e0994029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33065 } }
I20260620 02:30:39.212687 11769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:30:39.212832 11769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129255ec40944dd098734ec1e0994029, State: Initialized, Role: FOLLOWER
I20260620 02:30:39.213590 11769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129255ec40944dd098734ec1e0994029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33065 } }
I20260620 02:30:39.213936 11769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:30:39.214157 11769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:30:39.214375 11769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:30:39.218047 11769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129255ec40944dd098734ec1e0994029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33065 } }
I20260620 02:30:39.218694 11769 leader_election.cc:302] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129255ec40944dd098734ec1e0994029; no voters:
I20260620 02:30:39.219796 11769 leader_election.cc:288] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:30:39.219974 11774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:30:39.221410 11774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [term 1 LEADER]: Becoming Leader. State: Replica: 129255ec40944dd098734ec1e0994029, State: Running, Role: LEADER
I20260620 02:30:39.222184 11774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129255ec40944dd098734ec1e0994029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33065 } }
I20260620 02:30:39.223155 11769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:30:39.226106 11776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129255ec40944dd098734ec1e0994029. Latest consensus state: current_term: 1 leader_uuid: "129255ec40944dd098734ec1e0994029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129255ec40944dd098734ec1e0994029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33065 } } }
I20260620 02:30:39.227351 11776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [sys.catalog]: This master's current role is: LEADER
I20260620 02:30:39.228395 11775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "129255ec40944dd098734ec1e0994029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129255ec40944dd098734ec1e0994029" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33065 } } }
I20260620 02:30:39.230335 11775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029 [sys.catalog]: This master's current role is: LEADER
I20260620 02:30:39.238835 11780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:30:39.239136 11769 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:30:39.256413 11780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:30:39.278416 11780 catalog_manager.cc:1347] Generated new cluster ID: be5bef9c125f485fb73fb273f742b1e8
I20260620 02:30:39.278725 11780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:30:39.329654 11780 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:30:39.332783 11780 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:30:39.355610 11780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029: Generated new TSK 0
I20260620 02:30:39.357270 11780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:30:40.131752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33065
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:30:40 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:30:40.654821 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:30:40.655212 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:30:40.655411 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:30:40.666673 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:30:40.666939 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:30:40.667265 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:30:40.680588 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:45225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:30:40.682794 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:30:40.684414 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:30:40.725152 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:40.726856 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:30:40.730063 11814 server_base.cc:1034] running on GCE node
W20260620 02:30:40.730348 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:30:40.731213 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:30:40.732527 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:30:40.733906 11814 hybrid_clock.cc:648] HybridClock initialized: now 1781922640733758 us; error 106 us; skew 500 ppm
I20260620 02:30:40.734338 11814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:30:40 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:30:40.738322 11814 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:30:40.746337 11814 webserver.cc:460] Webserver started at http://127.0.97.1:42763/ using document root <none> and password file <none>
I20260620 02:30:40.747193 11814 fs_manager.cc:359] Metadata directory not provided
I20260620 02:30:40.747393 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:30:40.747807 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:30:40 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:30:40.768275 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "0d2eddc285244cd8b284367dbb6332c2"
format_stamp: "Formatted at 2026-06-20 02:30:40 on dist-test-slave-r706"
server_key: "a2f840e2001455ea0b0a95d936f9dfc6"
server_key_iv: "0283d549ce23866d1a52bea443f1ba69"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:40.771575 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0d2eddc285244cd8b284367dbb6332c2"
format_stamp: "Formatted at 2026-06-20 02:30:40 on dist-test-slave-r706"
server_key: "a2f840e2001455ea0b0a95d936f9dfc6"
server_key_iv: "0283d549ce23866d1a52bea443f1ba69"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:40.791849 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.015s sys 0.003s
I20260620 02:30:40.828358 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:40.831025 11814 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.010s sys 0.000s
I20260620 02:30:40.831455 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "0d2eddc285244cd8b284367dbb6332c2"
format_stamp: "Formatted at 2026-06-20 02:30:40 on dist-test-slave-r706"
server_key: "a2f840e2001455ea0b0a95d936f9dfc6"
server_key_iv: "0283d549ce23866d1a52bea443f1ba69"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:40.831976 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:30:40.921689 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:30:40.934372 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:30:40.935189 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:30:40.947180 11814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:30:40.972849 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:30:40.973172 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.001s
I20260620 02:30:40.973508 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:30:40.973718 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:30:41 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:30:41.288710 11814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45853
I20260620 02:30:41.288748 11946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45853 every 8 connection(s)
I20260620 02:30:41.290436 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:30:41.300383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 11814
I20260620 02:30:41.300623 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:30:41.301838 388 external_mini_cluster.cc:1428] Setting key 88d26ac82a3e7fc02120bff31cd3f5ec
Jun 20 02:30:41 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:30:41.327908 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:41.296512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47331 (local address 127.0.97.62:33065)
0620 02:30:41.303662 (+ 7150us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:41.303669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:41.307264 (+ 3595us) server_negotiation.cc:407] Connection header received
0620 02:30:41.307333 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:41.307339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:41.307424 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:41.307625 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:41.310301 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:41.312461 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:41.315813 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:41.316372 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:41.316527 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:41.320901 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:41.320907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:41.320910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:41.320952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:41.324335 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:41.325163 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:41.325171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:41.325176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:41.325345 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:41.325857 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:41.325867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:41.325869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:41.326223 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:41.326395 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:41.327158 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 02:30:41.327421 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6967,"thread_start_us":6828,"threads_started":1}
I20260620 02:30:41.329632 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:41.295498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33065 (local address 127.0.97.1:47331)
0620 02:30:41.304235 (+ 8737us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:41.304288 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:41.306881 (+ 2593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:41.307937 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:41.307953 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:41.308820 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:30:41.310002 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:41.310035 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:41.312742 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:41.312751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:41.315637 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:41.315652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:41.316709 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:41.316719 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:41.316982 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:41.317842 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:30:41.317882 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:30:41.320530 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:30:41.324479 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:30:41.324489 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:30:41.324508 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:30:41.324976 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:30:41.325465 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:30:41.325470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:30:41.325474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:30:41.325731 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:30:41.326356 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:30:41.326365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:30:41.326747 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 02:30:41.327051 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 02:30:41.328362 (+ 1311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8404,"thread_start_us":8162,"threads_started":1}
I20260620 02:30:41.331387 11947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33065
I20260620 02:30:41.331734 11947 heartbeater.cc:464] Registering TS with master...
I20260620 02:30:41.332635 11947 heartbeater.cc:511] Master 127.0.97.62:33065 requested a full tablet report, sending...
I20260620 02:30:41.335147 11734 ts_manager.cc:194] Registered new tserver with Master: 0d2eddc285244cd8b284367dbb6332c2 (127.0.97.1:45853)
I20260620 02:30:41.336896 11734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47331
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:30:41.380730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33065
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:30:41 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:30:42.076419 11953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:30:42.076838 11953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:30:42.077107 11953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:30:42.086454 11953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:30:42.086628 11953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:30:42.086825 11953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:30:42.099344 11953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:45225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:30:42.100880 11953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:30:42.102355 11953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:30:42.122668 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:42.123273 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:42.123533 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:30:42.125051 11953 server_base.cc:1034] running on GCE node
I20260620 02:30:42.125867 11953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:30:42.127074 11953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:30:42.128348 11953 hybrid_clock.cc:648] HybridClock initialized: now 1781922642128290 us; error 45 us; skew 500 ppm
I20260620 02:30:42.128731 11953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:30:42 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:30:42.134222 11953 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:30:42.136514 11953 webserver.cc:460] Webserver started at http://127.0.97.2:43815/ using document root <none> and password file <none>
I20260620 02:30:42.137614 11953 fs_manager.cc:359] Metadata directory not provided
I20260620 02:30:42.137814 11953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:30:42.138268 11953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:30:42 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:30:42.158025 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "bb8b45399fe64a368790250713e6a2c4"
format_stamp: "Formatted at 2026-06-20 02:30:42 on dist-test-slave-r706"
server_key: "deab9d95cc31b7e15754b94b4c4d9eed"
server_key_iv: "b133595a2d09da7a2f32edabe70e7ef4"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:42.159373 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bb8b45399fe64a368790250713e6a2c4"
format_stamp: "Formatted at 2026-06-20 02:30:42 on dist-test-slave-r706"
server_key: "deab9d95cc31b7e15754b94b4c4d9eed"
server_key_iv: "b133595a2d09da7a2f32edabe70e7ef4"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:42.166239 11953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 02:30:42.188506 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:42.190752 11953 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260620 02:30:42.190994 11953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "bb8b45399fe64a368790250713e6a2c4"
format_stamp: "Formatted at 2026-06-20 02:30:42 on dist-test-slave-r706"
server_key: "deab9d95cc31b7e15754b94b4c4d9eed"
server_key_iv: "b133595a2d09da7a2f32edabe70e7ef4"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:42.191309 11953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:30:42.212361 11953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:30:42.218801 11953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:30:42.219285 11953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:30:42.220918 11953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:30:42.223453 11953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:30:42.223582 11953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:42.223775 11953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:30:42.223865 11953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:42.342803 11947 heartbeater.cc:503] Master 127.0.97.62:33065 was elected leader, sending a full tablet report...
I20260620 02:30:42.469594 11953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35037
I20260620 02:30:42.469552 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35037 every 8 connection(s)
I20260620 02:30:42.477721 11953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:30:42.486150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 11953
I20260620 02:30:42.487401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:30:42.491097 388 external_mini_cluster.cc:1428] Setting key f481b7bfe61b9dcb7d7e93616667b4c7
Jun 20 02:30:42 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:30:42.742501 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:42.499602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41827 (local address 127.0.97.62:33065)
0620 02:30:42.508289 (+ 8687us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:42.508324 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:42.551772 (+ 43448us) server_negotiation.cc:407] Connection header received
0620 02:30:42.596321 (+ 44549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:42.596347 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:42.597232 (+ 885us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:42.597802 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:42.620159 (+ 22357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:42.625159 (+ 5000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:42.639978 (+ 14819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:42.642233 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:42.642769 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:42.670084 (+ 27315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:42.670131 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:42.670175 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:42.670588 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:42.705585 (+ 34997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:42.715292 (+ 9707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:42.715358 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:42.715414 (+ 56us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:42.717562 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:42.722775 (+ 5213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:42.722836 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:42.722878 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:42.726716 (+ 3838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:42.727617 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:42.736846 (+ 9229us) server_negotiation.cc:299] Negotiation successful
0620 02:30:42.739792 (+ 2946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7734,"thread_start_us":38617,"threads_started":1}
I20260620 02:30:42.746735 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:42.505161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33065 (local address 127.0.97.2:41827)
0620 02:30:42.550384 (+ 45223us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:42.550865 (+ 481us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:42.593831 (+ 42966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:42.602314 (+ 8483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:42.602433 (+ 119us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:42.609326 (+ 6893us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:30:42.617895 (+ 8569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:42.618004 (+ 109us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:42.626123 (+ 8119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:42.626144 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:42.638600 (+ 12456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:42.638862 (+ 262us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:42.643402 (+ 4540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:42.643468 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:42.644627 (+ 1159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:42.648095 (+ 3468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:30:42.648323 (+ 228us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:30:42.668073 (+ 19750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:30:42.707267 (+ 39194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:30:42.707342 (+ 75us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:30:42.707520 (+ 178us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:30:42.714287 (+ 6767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:30:42.718660 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:30:42.718714 (+ 54us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:30:42.718764 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:30:42.721641 (+ 2877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:30:42.727727 (+ 6086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:30:42.727758 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:30:42.731432 (+ 3674us) client_negotiation.cc:769] Sending connection context
0620 02:30:42.735469 (+ 4037us) client_negotiation.cc:240] Negotiation successful
0620 02:30:42.740253 (+ 4784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42320,"thread_start_us":66997,"threads_started":1}
I20260620 02:30:42.760763 12090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33065
I20260620 02:30:42.762487 12090 heartbeater.cc:464] Registering TS with master...
I20260620 02:30:42.764286 12090 heartbeater.cc:511] Master 127.0.97.62:33065 requested a full tablet report, sending...
I20260620 02:30:42.771281 11734 ts_manager.cc:194] Registered new tserver with Master: bb8b45399fe64a368790250713e6a2c4 (127.0.97.2:35037)
I20260620 02:30:42.776893 11734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260620 02:30:42.835074 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.276s user 0.003s sys 0.100s
I20260620 02:30:42.839639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33065
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:30:43.472285 12096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:30:43.472648 12096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:30:43.472916 12096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:30:43.482309 12096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:30:43.482501 12096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:30:43.482726 12096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:30:43.495029 12096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:45225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:30:43.496587 12096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:30:43.498193 12096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:30:43.516006 12101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:43.516006 12104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:30:43.516212 12102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:30:43.519999 12096 server_base.cc:1034] running on GCE node
I20260620 02:30:43.523597 12096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:30:43.528752 12096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:30:43.533286 12096 hybrid_clock.cc:648] HybridClock initialized: now 1781922643532752 us; error 252 us; skew 500 ppm
I20260620 02:30:43.535331 12096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:30:43 dist-test-slave-r706 krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:30:43.555744 12096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:30:43.566253 12096 webserver.cc:460] Webserver started at http://127.0.97.3:35239/ using document root <none> and password file <none>
I20260620 02:30:43.574671 12096 fs_manager.cc:359] Metadata directory not provided
I20260620 02:30:43.576397 12096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:30:43.578763 12096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:30:43 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:30:43.664321 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "b648a33d134a47e1911d7f0c601e6d44"
format_stamp: "Formatted at 2026-06-20 02:30:43 on dist-test-slave-r706"
server_key: "5044e2ddc702d0350d81f05dbc6df033"
server_key_iv: "492f09b17b9c58eec0e5395132ca06a1"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:43.669450 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b648a33d134a47e1911d7f0c601e6d44"
format_stamp: "Formatted at 2026-06-20 02:30:43 on dist-test-slave-r706"
server_key: "5044e2ddc702d0350d81f05dbc6df033"
server_key_iv: "492f09b17b9c58eec0e5395132ca06a1"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:43.704262 12096 fs_manager.cc:691] Time spent creating directory manager: real 0.032s user 0.019s sys 0.010s
I20260620 02:30:43.799201 12090 heartbeater.cc:503] Master 127.0.97.62:33065 was elected leader, sending a full tablet report...
I20260620 02:30:43.803879 12111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:43.814407 12096 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.016s sys 0.002s
I20260620 02:30:43.815409 12096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "b648a33d134a47e1911d7f0c601e6d44"
format_stamp: "Formatted at 2026-06-20 02:30:43 on dist-test-slave-r706"
server_key: "5044e2ddc702d0350d81f05dbc6df033"
server_key_iv: "492f09b17b9c58eec0e5395132ca06a1"
server_key_version: "kuduclusterkey@0"
I20260620 02:30:43.817030 12096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:30:43.853088 12096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:30:43.858850 12096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:30:43.859288 12096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:30:43.860769 12096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:30:43.863243 12096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:30:43.863389 12096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:43.863600 12096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:30:43.863708 12096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:30:43.933174 12096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37435
I20260620 02:30:43.933243 12223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37435 every 8 connection(s)
I20260620 02:30:43.934883 12096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:30:43.939530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 12096
I20260620 02:30:43.939738 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:30:43.940368 388 external_mini_cluster.cc:1428] Setting key 7a6ec8f7ed28fa1f27abda779647da19
Jun 20 02:30:43 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:30:43.963137 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:43.938280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56613 (local address 127.0.97.62:33065)
0620 02:30:43.939347 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:43.939356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:43.940537 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 02:30:43.944332 (+ 3795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:43.944336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:43.944434 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:43.944536 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:43.946453 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:43.947407 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:43.949814 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:43.950387 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:43.950527 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:43.955688 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:43.955694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:43.955697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:43.955744 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:43.959555 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:43.960425 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:43.960430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:43.960432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:43.960578 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:43.961170 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:43.961173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:43.961175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:43.961551 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:43.961685 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:43.962536 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 02:30:43.962791 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":528,"threads_started":1}
I20260620 02:30:43.964257 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:43.938776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33065 (local address 127.0.97.3:56613)
0620 02:30:43.940232 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:43.940295 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:43.944040 (+ 3745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:43.944760 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:43.944768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:43.945500 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:30:43.946283 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:43.946302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:43.947596 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:43.947600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:43.949612 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:43.949655 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:43.950751 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:43.950759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:43.951022 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:43.951807 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:30:43.951860 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:30:43.955285 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:30:43.959733 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:30:43.959741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:30:43.959756 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:30:43.960301 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:30:43.960700 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:30:43.960704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:30:43.960707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:30:43.960989 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:30:43.961723 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:30:43.961729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:30:43.962113 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 02:30:43.962411 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 02:30:43.962915 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":763,"threads_started":1}
I20260620 02:30:43.966017 12224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33065
I20260620 02:30:43.966403 12224 heartbeater.cc:464] Registering TS with master...
I20260620 02:30:43.967334 12224 heartbeater.cc:511] Master 127.0.97.62:33065 requested a full tablet report, sending...
I20260620 02:30:43.969458 11734 ts_manager.cc:194] Registered new tserver with Master: b648a33d134a47e1911d7f0c601e6d44 (127.0.97.3:37435)
I20260620 02:30:43.970139 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:30:43.970834 11734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56613
I20260620 02:30:43.992573 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:43.976199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.97.62:33065)
0620 02:30:43.976474 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:43.976484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:43.977309 (+ 825us) server_negotiation.cc:407] Connection header received
0620 02:30:43.977482 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:43.977485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:43.977585 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:43.977673 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:43.979493 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:43.980328 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:43.982117 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:43.982773 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:43.982877 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:43.986114 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:43.986119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:43.986122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:43.986157 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:43.989550 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:43.990333 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:43.990336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:43.990339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:43.990507 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:43.991011 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:43.991016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:43.991018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:43.991376 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:43.991625 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:43.991941 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:30:43.992132 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:30:43.997558 11734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:30:44.007630 11734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:30:44.057109 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.026354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.97.3:37435)
0620 02:30:44.042512 (+ 16158us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.042519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.043735 (+ 1216us) server_negotiation.cc:407] Connection header received
0620 02:30:44.043904 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.043915 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.044230 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.045219 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:30:44.045862 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.048437 (+ 2575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.052562 (+ 4125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.054801 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.054933 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.055285 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.056241 (+ 956us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.056475 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15960,"thread_start_us":16122,"threads_started":1}
I20260620 02:30:44.057432 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.026023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37435 (local address 127.0.0.1:47434)
0620 02:30:44.043145 (+ 17122us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:44.043229 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:44.043499 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:44.044528 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:44.044536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:44.044583 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:30:44.045498 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.045511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.048634 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.048639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.052312 (+ 3673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.052327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.055590 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.055598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.055868 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.055942 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 02:30:44.056161 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 02:30:44.056430 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16726,"spinlock_wait_cycles":14976,"thread_start_us":15512,"threads_started":1}
I20260620 02:30:44.059154 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.027033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.2:35037)
0620 02:30:44.041246 (+ 14213us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.041259 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.044102 (+ 2843us) server_negotiation.cc:407] Connection header received
0620 02:30:44.044279 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.044301 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.044589 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.044891 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:30:44.046786 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.049072 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.053424 (+ 4352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.055563 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.055726 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.056064 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.058350 (+ 2286us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.058594 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13873,"thread_start_us":13597,"threads_started":1}
I20260620 02:30:44.060173 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.026325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35037 (local address 127.0.0.1:50032)
0620 02:30:44.043145 (+ 16820us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:44.043229 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:44.043499 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:44.045487 (+ 1988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:44.045491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:44.045518 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:30:44.046440 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.046460 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.049923 (+ 3463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.049928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.053214 (+ 3286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.053230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.056319 (+ 3089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.056326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.056779 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.056814 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:30:44.056896 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 02:30:44.059415 (+ 2519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16543,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":14975,"threads_started":1}
I20260620 02:30:44.060920 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.042691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45853 (local address 127.0.0.1:41962)
0620 02:30:44.050029 (+ 7338us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:44.050061 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:44.050282 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:44.051240 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:44.051245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:44.051274 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:30:44.052201 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.052216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.054622 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.054627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.056806 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.056829 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.059635 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.059643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.059919 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.059968 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:30:44.060142 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 02:30:44.060324 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7139,"mutex_wait_us":1013,"thread_start_us":5901,"threads_started":1}
I20260620 02:30:44.061033 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.043364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.97.1:45853)
0620 02:30:44.044809 (+ 1445us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.044815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.050233 (+ 5418us) server_negotiation.cc:407] Connection header received
0620 02:30:44.050620 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.050635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.050970 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.051290 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:30:44.052556 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.054433 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.057080 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.059187 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.059387 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.059696 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.060223 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.060483 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":996,"threads_started":1}
I20260620 02:30:44.063009 12159 tablet_service.cc:1467] Processing CreateTablet for tablet f25c34076398466c9f244cb5fadf1bda (DEFAULT_TABLE table=test-table [id=964e2ebc75e8425b8183aba2e497e6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:30:44.064642 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25c34076398466c9f244cb5fadf1bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:30:44.065709 12025 tablet_service.cc:1467] Processing CreateTablet for tablet f25c34076398466c9f244cb5fadf1bda (DEFAULT_TABLE table=test-table [id=964e2ebc75e8425b8183aba2e497e6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:30:44.065627 11882 tablet_service.cc:1467] Processing CreateTablet for tablet f25c34076398466c9f244cb5fadf1bda (DEFAULT_TABLE table=test-table [id=964e2ebc75e8425b8183aba2e497e6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:30:44.068336 11882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25c34076398466c9f244cb5fadf1bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:30:44.068331 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25c34076398466c9f244cb5fadf1bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:30:44.081660 12243 tablet_bootstrap.cc:492] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Bootstrap starting.
I20260620 02:30:44.087868 12243 tablet_bootstrap.cc:654] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Neither blocks nor log segments found. Creating new log.
I20260620 02:30:44.089563 12243 log.cc:824] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Log is configured to *not* fsync() on all Append() calls
I20260620 02:30:44.091930 12245 tablet_bootstrap.cc:492] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Bootstrap starting.
I20260620 02:30:44.094193 12246 tablet_bootstrap.cc:492] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Bootstrap starting.
I20260620 02:30:44.094745 12243 tablet_bootstrap.cc:492] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: No bootstrap required, opened a new log
I20260620 02:30:44.095206 12243 ts_tablet_manager.cc:1397] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260620 02:30:44.099620 12246 tablet_bootstrap.cc:654] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:30:44.099956 12245 tablet_bootstrap.cc:654] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Neither blocks nor log segments found. Creating new log.
I20260620 02:30:44.101369 12246 log.cc:824] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:30:44.102074 12245 log.cc:824] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:30:44.106066 12245 tablet_bootstrap.cc:492] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: No bootstrap required, opened a new log
I20260620 02:30:44.106444 12245 ts_tablet_manager.cc:1397] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260620 02:30:44.106586 12246 tablet_bootstrap.cc:492] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: No bootstrap required, opened a new log
I20260620 02:30:44.107061 12246 ts_tablet_manager.cc:1397] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260620 02:30:44.111550 12243 raft_consensus.cc:348] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.112216 12243 raft_consensus.cc:374] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:30:44.112414 12243 raft_consensus.cc:729] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b648a33d134a47e1911d7f0c601e6d44, State: Initialized, Role: FOLLOWER
I20260620 02:30:44.113384 12243 consensus_queue.cc:260] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.115267 12224 heartbeater.cc:503] Master 127.0.97.62:33065 was elected leader, sending a full tablet report...
I20260620 02:30:44.116148 12243 ts_tablet_manager.cc:1428] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260620 02:30:44.119109 12245 raft_consensus.cc:348] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.119653 12245 raft_consensus.cc:374] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:30:44.119812 12245 raft_consensus.cc:729] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2eddc285244cd8b284367dbb6332c2, State: Initialized, Role: FOLLOWER
I20260620 02:30:44.120566 12245 consensus_queue.cc:260] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.121924 12246 raft_consensus.cc:348] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.122542 12246 raft_consensus.cc:374] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:30:44.122701 12246 raft_consensus.cc:729] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8b45399fe64a368790250713e6a2c4, State: Initialized, Role: FOLLOWER
I20260620 02:30:44.123329 12245 ts_tablet_manager.cc:1428] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Time spent starting tablet: real 0.017s user 0.004s sys 0.014s
I20260620 02:30:44.123564 12246 consensus_queue.cc:260] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.124459 12249 raft_consensus.cc:482] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:30:44.124945 12249 raft_consensus.cc:504] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.126329 12246 ts_tablet_manager.cc:1428] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260620 02:30:44.127566 12249 leader_election.cc:288] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d2eddc285244cd8b284367dbb6332c2 (127.0.97.1:45853), bb8b45399fe64a368790250713e6a2c4 (127.0.97.2:35037)
I20260620 02:30:44.137845 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.129375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43903 (local address 127.0.97.1:45853)
0620 02:30:44.129614 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.129620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.129641 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:30:44.129703 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.129708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.129835 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.129932 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:30:44.130448 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.132187 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.135076 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.136761 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.136936 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.137118 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.137509 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.137613 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:30:44.138021 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.128417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45853 (local address 127.0.97.3:43903)
0620 02:30:44.128651 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:44.128673 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:44.128830 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:44.129949 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:44.129954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:44.129985 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:30:44.130267 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.130279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.132418 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.132423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.134868 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.134881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.136971 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.136978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.137306 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.137352 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:30:44.137464 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:30:44.137625 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:30:44.139806 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.129196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44135 (local address 127.0.97.2:35037)
0620 02:30:44.129547 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.129564 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.130635 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 02:30:44.130825 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.130831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.130931 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.131069 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:30:44.132358 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.134040 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.136977 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.138714 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.138875 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.138985 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.139460 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.139563 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:30:44.140202 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.128947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35037 (local address 127.0.97.3:44135)
0620 02:30:44.130484 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0620 02:30:44.130508 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:30:44.130645 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:30:44.131312 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:30:44.131317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:30:44.131346 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:30:44.132114 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.132130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.134245 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.134249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.136780 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:30:44.136796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.139041 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.139051 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:30:44.139301 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.139334 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:30:44.139439 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 02:30:44.139578 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":1137,"threads_started":1}
I20260620 02:30:44.139689 11902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25c34076398466c9f244cb5fadf1bda" candidate_uuid: "b648a33d134a47e1911d7f0c601e6d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2eddc285244cd8b284367dbb6332c2" is_pre_election: true
I20260620 02:30:44.140717 11902 raft_consensus.cc:2454] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b648a33d134a47e1911d7f0c601e6d44 in term 0.
I20260620 02:30:44.141574 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25c34076398466c9f244cb5fadf1bda" candidate_uuid: "b648a33d134a47e1911d7f0c601e6d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8b45399fe64a368790250713e6a2c4" is_pre_election: true
I20260620 02:30:44.141919 12114 leader_election.cc:302] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2eddc285244cd8b284367dbb6332c2, b648a33d134a47e1911d7f0c601e6d44; no voters:
I20260620 02:30:44.142478 12249 raft_consensus.cc:2781] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:30:44.142478 12045 raft_consensus.cc:2454] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b648a33d134a47e1911d7f0c601e6d44 in term 0.
I20260620 02:30:44.142753 12249 raft_consensus.cc:482] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:30:44.142930 12249 raft_consensus.cc:3037] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:30:44.146111 12249 raft_consensus.cc:504] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.147029 12249 leader_election.cc:288] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [CANDIDATE]: Term 1 election: Requested vote from peers 0d2eddc285244cd8b284367dbb6332c2 (127.0.97.1:45853), bb8b45399fe64a368790250713e6a2c4 (127.0.97.2:35037)
I20260620 02:30:44.147701 11902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25c34076398466c9f244cb5fadf1bda" candidate_uuid: "b648a33d134a47e1911d7f0c601e6d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2eddc285244cd8b284367dbb6332c2"
I20260620 02:30:44.147701 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25c34076398466c9f244cb5fadf1bda" candidate_uuid: "b648a33d134a47e1911d7f0c601e6d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8b45399fe64a368790250713e6a2c4"
I20260620 02:30:44.147980 12045 raft_consensus.cc:3037] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:30:44.147980 11902 raft_consensus.cc:3037] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:30:44.152776 11902 raft_consensus.cc:2454] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b648a33d134a47e1911d7f0c601e6d44 in term 1.
I20260620 02:30:44.152761 12045 raft_consensus.cc:2454] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b648a33d134a47e1911d7f0c601e6d44 in term 1.
I20260620 02:30:44.153421 12112 leader_election.cc:302] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d2eddc285244cd8b284367dbb6332c2, b648a33d134a47e1911d7f0c601e6d44, bb8b45399fe64a368790250713e6a2c4; no voters:
I20260620 02:30:44.154019 12249 raft_consensus.cc:2781] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:30:44.154729 12249 raft_consensus.cc:686] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 1 LEADER]: Becoming Leader. State: Replica: b648a33d134a47e1911d7f0c601e6d44, State: Running, Role: LEADER
I20260620 02:30:44.155340 12249 consensus_queue.cc:237] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } }
I20260620 02:30:44.160769 11733 catalog_manager.cc:5495] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 reported cstate change: term changed from 0 to 1, leader changed from <none> to b648a33d134a47e1911d7f0c601e6d44 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b648a33d134a47e1911d7f0c601e6d44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648a33d134a47e1911d7f0c601e6d44" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 } health_report { overall_health: UNKNOWN } } }
W20260620 02:30:44.187775 12225 tablet.cc:2369] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:30:44.190251 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.182698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.0.97.3:37435)
0620 02:30:44.182996 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.183004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.183095 (+ 91us) server_negotiation.cc:407] Connection header received
0620 02:30:44.183316 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.183320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.183406 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.183593 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:30:44.184044 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.184869 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.187210 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.188229 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.188369 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.189061 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:30:44.189606 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:30:44.189716 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.189866 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.189954 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 02:30:44.200192 11902 raft_consensus.cc:1261] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 1 FOLLOWER]: Refusing update from remote peer b648a33d134a47e1911d7f0c601e6d44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:30:44.200193 12045 raft_consensus.cc:1261] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 1 FOLLOWER]: Refusing update from remote peer b648a33d134a47e1911d7f0c601e6d44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:30:44.201344 12253 consensus_queue.cc:1035] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2eddc285244cd8b284367dbb6332c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:30:44.201828 12249 consensus_queue.cc:1035] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8b45399fe64a368790250713e6a2c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:30:44.218339 12257 mvcc.cc:205] Tried to move back new op lower bound from 7298755150631747584 to 7298755150464942080. Current Snapshot: MvccSnapshot[applied={T|T < 7298755150631747584}]
I20260620 02:30:44.235657 11734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:30:44.236070 11734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:30:44.240681 11734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 129255ec40944dd098734ec1e0994029: Sending DeleteTablet for 3 replicas of tablet f25c34076398466c9f244cb5fadf1bda
I20260620 02:30:44.242298 12025 tablet_service.cc:1514] Processing DeleteTablet for tablet f25c34076398466c9f244cb5fadf1bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:30:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50032
I20260620 02:30:44.242424 11882 tablet_service.cc:1514] Processing DeleteTablet for tablet f25c34076398466c9f244cb5fadf1bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:30:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41962
I20260620 02:30:44.243809 12159 tablet_service.cc:1514] Processing DeleteTablet for tablet f25c34076398466c9f244cb5fadf1bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:30:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47434
I20260620 02:30:44.244729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 11814
I20260620 02:30:44.249682 12266 tablet_replica.cc:330] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: stopping tablet replica
I20260620 02:30:44.250327 12266 raft_consensus.cc:2229] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:30:44.251099 12266 raft_consensus.cc:2258] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:30:44.252058 12267 tablet_replica.cc:330] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: stopping tablet replica
I20260620 02:30:44.252566 12267 raft_consensus.cc:2229] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:30:44.253470 12267 raft_consensus.cc:2258] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:30:44.256093 12266 ts_tablet_manager.cc:1905] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:30:44.264671 12266 ts_tablet_manager.cc:1918] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:30:44.264947 12266 log.cc:1196] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/f25c34076398466c9f244cb5fadf1bda
I20260620 02:30:44.267525 12267 ts_tablet_manager.cc:1905] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:30:44.270031 12265 tablet_replica.cc:330] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: stopping tablet replica
I20260620 02:30:44.270551 12265 raft_consensus.cc:2229] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:30:44.271082 12265 raft_consensus.cc:2258] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:30:44.273160 12265 ts_tablet_manager.cc:1905] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:30:44 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:30:44.278280 12267 ts_tablet_manager.cc:1918] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:30:44.278569 12267 log.cc:1196] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/f25c34076398466c9f244cb5fadf1bda
I20260620 02:30:44.279354 12267 ts_tablet_manager.cc:1939] T f25c34076398466c9f244cb5fadf1bda P b648a33d134a47e1911d7f0c601e6d44: Deleting consensus metadata
I20260620 02:30:44.281883 11720 catalog_manager.cc:4841] TS b648a33d134a47e1911d7f0c601e6d44 (127.0.97.3:37435): tablet f25c34076398466c9f244cb5fadf1bda (table test-table [id=964e2ebc75e8425b8183aba2e497e6ec]) successfully deleted
I20260620 02:30:44.282315 12265 ts_tablet_manager.cc:1918] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:30:44.282524 12265 log.cc:1196] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/f25c34076398466c9f244cb5fadf1bda
I20260620 02:30:44.282874 12266 ts_tablet_manager.cc:1939] T f25c34076398466c9f244cb5fadf1bda P 0d2eddc285244cd8b284367dbb6332c2: Deleting consensus metadata
I20260620 02:30:44.283198 12265 ts_tablet_manager.cc:1939] T f25c34076398466c9f244cb5fadf1bda P bb8b45399fe64a368790250713e6a2c4: Deleting consensus metadata
I20260620 02:30:44.285398 11720 catalog_manager.cc:4841] TS 0d2eddc285244cd8b284367dbb6332c2 (127.0.97.1:45853): tablet f25c34076398466c9f244cb5fadf1bda (table test-table [id=964e2ebc75e8425b8183aba2e497e6ec]) successfully deleted
I20260620 02:30:44.285487 11718 catalog_manager.cc:4841] TS bb8b45399fe64a368790250713e6a2c4 (127.0.97.2:35037): tablet f25c34076398466c9f244cb5fadf1bda (table test-table [id=964e2ebc75e8425b8183aba2e497e6ec]) successfully deleted
I20260620 02:30:44.288687 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.246732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.97.1:45853)
0620 02:30:44.246982 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.246991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.257864 (+ 10873us) server_negotiation.cc:407] Connection header received
0620 02:30:44.258485 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.258494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.258607 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.258736 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:44.261323 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.262927 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.266652 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.267562 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.267692 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.278918 (+ 11226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:44.278942 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:44.278963 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:44.279032 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:44.284609 (+ 5577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:44.286141 (+ 1532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:44.286149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:44.286152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:44.286285 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:44.286756 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:44.286781 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:44.286786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:44.287256 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:44.287625 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.287915 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.288187 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:30:44.289645 11942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:30:44.419286 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 11814
I20260620 02:30:44.438688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 11953
Jun 20 02:30:44 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:30:44.455658 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.439586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.0.97.2:35037)
0620 02:30:44.439777 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.439782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.439802 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:30:44.440015 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.440019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.440089 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.440211 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:44.441848 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.442591 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.444050 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.444481 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.444625 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.449277 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:44.449293 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:44.449309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:44.449350 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:44.452534 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:44.453416 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:44.453423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:44.453426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:44.453565 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:44.453977 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:44.453986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:44.453989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:44.454373 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:44.454730 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.455036 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.455278 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:30:44.456588 12085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:30:44.587113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 11953
I20260620 02:30:44.606148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 12096
Jun 20 02:30:44 dist-test-slave-r706 krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:30:44.626175 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:30:44.607200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.0.97.3:37435)
0620 02:30:44.607422 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:30:44.607437 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:30:44.607462 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:30:44.607546 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:30:44.607552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:30:44.607641 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:30:44.607770 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:30:44.609131 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.610416 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.611908 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:30:44.612620 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:30:44.612774 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:30:44.618059 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:30:44.618076 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:30:44.618098 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:30:44.618176 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:30:44.622846 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:44.623695 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:44.623709 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:44.623714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:44.623943 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:30:44.624341 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:30:44.624346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:30:44.624350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:30:44.624825 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:30:44.625194 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 02:30:44.625512 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:30:44.625767 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:30:44.627074 12219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:30:44.781564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 12096
I20260620 02:30:44.803324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 11702
I20260620 02:30:44.804682 11764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:30:44.959767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 11702
I20260620 02:30:44.985615 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:30:45.322239 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:30:45.322481 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:30:45.668877 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:30:45.669 UTC [11288] LOG: received smart shutdown request
2026-06-20 02:30:45.673 UTC [11288] LOG: background worker "logical replication launcher" (PID 11297) exited with exit code 1
2026-06-20 02:30:45.673 UTC [11292] LOG: shutting down
2026-06-20 02:30:45.850 UTC [11288] LOG: database system is shut down
2026-06-20T02:30:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54037 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:30:46 dist-test-slave-r706 krb5kdc[12281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:30:46 dist-test-slave-r706 krb5kdc[12281](info): set up 2 sockets
Jun 20 02:30:46 dist-test-slave-r706 krb5kdc[12281](info): commencing operation
krb5kdc: starting...
W20260620 02:30:47.105463 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:30:47 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:30:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 02:30:47.472154 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkyewzx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/postgres -l logfile start
2026-06-20 02:30:50.241 UTC [12311] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:30:50.241 UTC [12311] LOG: listening on IPv4 address "127.0.97.20", port 54971
2026-06-20 02:30:50.245 UTC [12311] LOG: listening on Unix socket "/tmp/.s.PGSQL.54971"
2026-06-20 02:30:50.267 UTC [12314] LOG: database system was shut down at 2026-06-20 02:30:48 UTC
2026-06-20 02:30:50.271 UTC [12311] LOG: database system is ready to accept connections
I20260620 02:30:51.249401 388 mini_postgres.cc:96] Postgres bound to 54971
2026-06-20 02:30:51.268 UTC [12322] FATAL: database "slave" does not exist
127.0.97.20:54971 - accepting connections
I20260620 02:30:51.269649 388 mini_ranger.cc:161] Starting Ranger...
I20260620 02:30:51.298533 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:30:51.842708 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:30:51.842862 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-admin
2026-06-20 02:30:52,153 [I] DB FLAVOR :POSTGRES
2026-06-20 02:30:52,153 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:30:52,153 [I] Checking connection..
2026-06-20 02:30:52,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:30:52,410 [I] Checking connection passed.
2026-06-20 02:30:52,410 [I] --------- Verifying version history table ---------
2026-06-20 02:30:52,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:30:52,672 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:30:52,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:30:52,939 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:30:52,939 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:30:52,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:30:53.188 UTC [12386] WARNING: there is no transaction in progress
2026-06-20 02:30:53,195 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:30:53,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:30:53,430 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:30:53,430 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:30:53,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:30:53,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:30:53,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r706.c.gcp-upstream.internal','N') ;"
2026-06-20 02:30:54,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:30:54,449 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:30:54,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:30:54,700 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:30:54,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:30:54,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:30:55,230 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:30:55,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:30:56.274 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.293 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.311 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.329 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.346 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.368 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.432 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.439 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.458 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.472 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.486 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.507 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.523 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.533 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.542 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:56.552 UTC [12494] WARNING: there is no transaction in progress
2026-06-20 02:30:57,308 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:30:57,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:30:57,540 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:30:57,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:30:57,792 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:30:57,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:30:58,061 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:30:58,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:30:58,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:30:58,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r706.c.gcp-upstream.internal';"
2026-06-20 02:30:58,857 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:30:58,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:30:59,125 [I] DB_PATCHES have already been applied
I20260620 02:30:59.131671 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkyewzx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:30:59.131812 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 02:30:59.148455 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:36111
Jun 20, 2026 2:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:30:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:31:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:31:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:31:00.320060 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.050s user 0.002s sys 0.047s
Jun 20, 2026 2:31:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:31:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 2:31:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:31:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:31:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:31:22.745373 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:31:22 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922682, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:31:22.937857 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:31:22.964552 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:31:24.042623 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:31:24.042762 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms
I20260620 02:31:24.042817 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54971
2026-06-20 02:31:24,437 [I] DB FLAVOR :POSTGRES
2026-06-20 02:31:24,437 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:31:24,437 [I] Checking connection
2026-06-20 02:31:24,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:31:24,725 [I] connection success
2026-06-20 02:31:24,726 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:31:24,726 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:31:24,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:31:24,997 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:31:24,997 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:31:24,997 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:31:24,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:31:25,311 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:31:25.879453 388 mini_ranger_kms.cc:310] Created kms service
I20260620 02:31:26.107062 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:31:26.158552 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:31:26 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922682, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:26.643236 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:31:26.643594 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkyewzx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkyewzx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:31:26.643657 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 02:31:26.662818 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12675
I20260620 02:31:26.662976 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:57769
Jun 20, 2026 2:31:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:31:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:31:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:31:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:31:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:31:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 02:31:28.926343 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.988s user 0.072s sys 0.344s
I20260620 02:31:28.926601 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:31:28.926781 388 mini_ranger_kms.cc:401] 127.0.97.20:57769/kms/v1/keys
Jun 20 02:31:29 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:31:29 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:31:29 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922689, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:31:32 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:31:32.453140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35481
--ranger_config_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:32.819041 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:32.819378 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:32.819566 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:32.827798 12725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:32.827951 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:32.828048 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:32.828147 12725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:32.828233 12725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:32.840502 12725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:57769/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35481
--ranger_config_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:32.841990 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:32.843400 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:32.856880 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:32.856911 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:32.856946 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:32.857570 12725 server_base.cc:1034] running on GCE node
I20260620 02:31:32.858551 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:32.860159 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:32.861449 12725 hybrid_clock.cc:648] HybridClock initialized: now 1781922692861402 us; error 61 us; skew 500 ppm
I20260620 02:31:32.861868 12725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:32 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:32.865291 12725 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:32.866854 12725 webserver.cc:460] Webserver started at http://127.0.97.62:36335/ using document root <none> and password file <none>
I20260620 02:31:32.867588 12725 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:32.867735 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:32.868059 12725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:31:32 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:32.934219 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "07ae14bf2ce042de826aa8befddc5514"
format_stamp: "Formatted at 2026-06-20 02:31:32 on dist-test-slave-r706"
server_key: "77f4ebb5c5795c15ffb5d82aae5612f3"
server_key_iv: "ac511ebe94f2d87f95fd9e3f4f849b95"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:32.935441 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "07ae14bf2ce042de826aa8befddc5514"
format_stamp: "Formatted at 2026-06-20 02:31:32 on dist-test-slave-r706"
server_key: "77f4ebb5c5795c15ffb5d82aae5612f3"
server_key_iv: "ac511ebe94f2d87f95fd9e3f4f849b95"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:32.943105 12725 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260620 02:31:32.968154 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:32.970010 12725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:31:32.970252 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "07ae14bf2ce042de826aa8befddc5514"
format_stamp: "Formatted at 2026-06-20 02:31:32 on dist-test-slave-r706"
server_key: "77f4ebb5c5795c15ffb5d82aae5612f3"
server_key_iv: "ac511ebe94f2d87f95fd9e3f4f849b95"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:32.970528 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:32.986135 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:32.991171 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:32.991585 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:33.016583 12725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35481
I20260620 02:31:33.016687 12791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35481 every 8 connection(s)
I20260620 02:31:33.018115 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:33.022286 12792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:33.025676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 12725
I20260620 02:31:33.025935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:31:33.026615 388 external_mini_cluster.cc:1428] Setting key 5ddec19fef53763fd59ff200847c38d9
I20260620 02:31:33.037019 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Bootstrap starting.
I20260620 02:31:33.040911 12792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:33.042110 12792 log.cc:824] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:31:33 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:33.045746 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: No bootstrap required, opened a new log
I20260620 02:31:33.054347 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:33.028916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.97.62:35481)
0620 02:31:33.029906 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:33.029921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:33.029975 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:31:33.032446 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:33.032483 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:33.033069 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:33.033604 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:33.034920 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:33.036889 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:33.038368 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:33.039470 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:33.039668 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:33.045138 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:33.045156 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:33.045179 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:33.045242 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:33.049681 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:33.050672 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:33.050679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:33.050686 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:33.050853 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:33.051314 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:33.051318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:33.051320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:33.051997 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:33.052254 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:33.052787 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 02:31:33.053257 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":428,"threads_started":1}
I20260620 02:31:33.055939 12792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:33.056378 12792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:33.056506 12792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae14bf2ce042de826aa8befddc5514, State: Initialized, Role: FOLLOWER
I20260620 02:31:33.057197 12792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:33.057480 12792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:31:33.057611 12792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:31:33.057806 12792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:33.060423 12792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:33.060932 12792 leader_election.cc:302] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07ae14bf2ce042de826aa8befddc5514; no voters:
I20260620 02:31:33.061697 12792 leader_election.cc:288] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:31:33.061885 12797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:33.063102 12797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae14bf2ce042de826aa8befddc5514, State: Running, Role: LEADER
I20260620 02:31:33.063691 12797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:33.064568 12792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:33.067157 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07ae14bf2ce042de826aa8befddc5514. Latest consensus state: current_term: 1 leader_uuid: "07ae14bf2ce042de826aa8befddc5514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260620 02:31:33.067716 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:33.068424 12798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07ae14bf2ce042de826aa8befddc5514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260620 02:31:33.068787 12798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:33.070405 12806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:33.076464 12806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:33.079188 12792 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:31:33.098006 12806 catalog_manager.cc:1347] Generated new cluster ID: fce6d92c00bf4e62afd38d1d6c7899a6
I20260620 02:31:33.098192 12806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:31:33.127694 12806 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:31:33.129967 12806 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:31:33.143180 12806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Generated new TSK 0
I20260620 02:31:33.144610 12806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:31:33.981828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:31:34 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:31:34.450948 12837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:34.451398 12837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:34.451635 12837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:34.464207 12837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:34.464394 12837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:34.464583 12837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:31:34.481536 12837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:57769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:34.483482 12837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:34.485469 12837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:34.504227 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:34.510264 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:34.513386 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:34.514808 12837 server_base.cc:1034] running on GCE node
I20260620 02:31:34.515704 12837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:34.517261 12837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:34.518638 12837 hybrid_clock.cc:648] HybridClock initialized: now 1781922694518568 us; error 60 us; skew 500 ppm
I20260620 02:31:34.519089 12837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:34 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:34.523981 12837 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:31:34.526340 12837 webserver.cc:460] Webserver started at http://127.0.97.1:44403/ using document root <none> and password file <none>
I20260620 02:31:34.527420 12837 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:34.527622 12837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:34.528035 12837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:31:34 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:34.550334 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "331b6d7ce97344ad9751d951196b4e54"
format_stamp: "Formatted at 2026-06-20 02:31:34 on dist-test-slave-r706"
server_key: "0aa0b9a240d3e693db6160c60137f799"
server_key_iv: "de37a807da5c4e9f3791bb9be3a392f0"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:34.551751 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "331b6d7ce97344ad9751d951196b4e54"
format_stamp: "Formatted at 2026-06-20 02:31:34 on dist-test-slave-r706"
server_key: "0aa0b9a240d3e693db6160c60137f799"
server_key_iv: "de37a807da5c4e9f3791bb9be3a392f0"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:34.571661 12837 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.007s sys 0.008s
I20260620 02:31:34.597726 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:34.599876 12837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:31:34.600114 12837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "331b6d7ce97344ad9751d951196b4e54"
format_stamp: "Formatted at 2026-06-20 02:31:34 on dist-test-slave-r706"
server_key: "0aa0b9a240d3e693db6160c60137f799"
server_key_iv: "de37a807da5c4e9f3791bb9be3a392f0"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:34.600427 12837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:34.639206 12837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:34.644657 12837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:34.645061 12837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:34.646258 12837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:34.647934 12837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:34.648044 12837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:34.648187 12837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:34.648276 12837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:34.750954 12837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33665
I20260620 02:31:34.750995 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33665 every 8 connection(s)
I20260620 02:31:34.752969 12837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:31:34.756798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 12837
I20260620 02:31:34.757135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:31:34.757979 388 external_mini_cluster.cc:1428] Setting key 208a93886af9ccb9f14b4aec2b1dddb3
Jun 20 02:31:34 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:31:34.802126 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:34.757708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51641 (local address 127.0.97.62:35481)
0620 02:31:34.765830 (+ 8122us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:34.765837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:34.767781 (+ 1944us) server_negotiation.cc:407] Connection header received
0620 02:31:34.769276 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:34.769282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:34.769376 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:34.769479 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:34.771834 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:34.773065 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:34.775647 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:34.776327 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:34.776455 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:34.785947 (+ 9492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:34.785958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:34.785964 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:34.786021 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:34.790170 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:34.791416 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:34.791422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:34.791426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:34.791607 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:34.792276 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:34.792281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:34.792284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:34.792673 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:34.801174 (+ 8501us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:34.801419 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 02:31:34.801719 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7982,"thread_start_us":8677,"threads_started":1}
I20260620 02:31:34.803388 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:34.757059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.1:51641)
0620 02:31:34.766118 (+ 9059us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:34.766184 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:34.768801 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:34.769978 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:34.769991 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:34.770652 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:34.771588 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:34.771613 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:34.773291 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:34.773297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:34.775447 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:34.775463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:34.776861 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:34.776868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:34.777219 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:34.778274 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:34.778306 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:34.785467 (+ 7161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:34.790451 (+ 4984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:34.790465 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:34.790488 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:34.791225 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:34.791793 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:34.791799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:34.791803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:34.792111 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:34.793201 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:34.793211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:34.793691 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 02:31:34.794050 (+ 359us) client_negotiation.cc:240] Negotiation successful
0620 02:31:34.794665 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8310,"thread_start_us":8031,"threads_started":1}
I20260620 02:31:34.805426 12970 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260620 02:31:34.805817 12970 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:34.806807 12970 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:34.809659 12757 ts_manager.cc:194] Registered new tserver with Master: 331b6d7ce97344ad9751d951196b4e54 (127.0.97.1:33665)
I20260620 02:31:34.811940 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51641
Jun 20 02:31:34 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:31:34.848959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:31:35 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:31:35.360785 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:35.361263 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:35.361433 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:35.373960 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:35.374199 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:35.374446 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:31:35.391176 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:57769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:35.393244 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:35.395071 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:35.412483 12983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:35.413080 12984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:35.422446 12986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:35.423107 12976 server_base.cc:1034] running on GCE node
I20260620 02:31:35.423867 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:35.425343 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:35.426728 12976 hybrid_clock.cc:648] HybridClock initialized: now 1781922695426559 us; error 118 us; skew 500 ppm
I20260620 02:31:35.427660 12976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:35 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:35.432762 12976 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:31:35.453504 12976 webserver.cc:460] Webserver started at http://127.0.97.2:35885/ using document root <none> and password file <none>
I20260620 02:31:35.454380 12976 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:35.454533 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:35.454931 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:31:35 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:35.471757 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "8bb70000273943cba13ac7c016f1fcc8"
format_stamp: "Formatted at 2026-06-20 02:31:35 on dist-test-slave-r706"
server_key: "e2f1506e4adf7d909f7960f098900472"
server_key_iv: "79b1c99ad56e9476f8fe5650b51a6d30"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:35.472680 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8bb70000273943cba13ac7c016f1fcc8"
format_stamp: "Formatted at 2026-06-20 02:31:35 on dist-test-slave-r706"
server_key: "e2f1506e4adf7d909f7960f098900472"
server_key_iv: "79b1c99ad56e9476f8fe5650b51a6d30"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:35.478768 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 02:31:35.493214 13000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:35.494951 12976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:31:35.495229 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "8bb70000273943cba13ac7c016f1fcc8"
format_stamp: "Formatted at 2026-06-20 02:31:35 on dist-test-slave-r706"
server_key: "e2f1506e4adf7d909f7960f098900472"
server_key_iv: "79b1c99ad56e9476f8fe5650b51a6d30"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:35.495510 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:35.508873 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:35.513342 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:35.513690 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:35.514742 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:35.516180 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:35.516304 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:35.516465 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:35.516561 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:35.553684 12976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37421
I20260620 02:31:35.553776 13112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37421 every 8 connection(s)
I20260620 02:31:35.555068 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:31:35.564230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 12976
I20260620 02:31:35.564460 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:31:35.565086 388 external_mini_cluster.cc:1428] Setting key c8db7a4460f557bab5534adab2ba2e58
Jun 20 02:31:35 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:35.577412 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:35.557611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34585 (local address 127.0.97.62:35481)
0620 02:31:35.558118 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:35.558124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:35.559114 (+ 990us) server_negotiation.cc:407] Connection header received
0620 02:31:35.560954 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:35.560958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:35.561106 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:35.561208 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:35.562628 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:35.563408 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:35.566026 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:35.566541 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:35.566648 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:35.570954 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:35.570959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:35.570962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:35.571011 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:35.573816 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:35.574613 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:35.574619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:35.574622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:35.574751 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:35.575253 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:35.575261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:35.575267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:35.575719 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:35.575897 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:35.576874 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 02:31:35.577139 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260620 02:31:35.578970 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:35.558125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.2:34585)
0620 02:31:35.558952 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:35.558991 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:35.560688 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:35.561418 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:35.561427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:35.561865 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:35.562454 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:35.562471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:35.563707 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:35.563712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:35.565889 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:35.565903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:35.566803 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:35.566811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:35.567116 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:35.567865 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:35.567897 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:35.570682 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:35.573980 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:35.573990 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:35.574010 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:35.574486 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:35.574876 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:35.574881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:35.574885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:35.575075 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:35.575907 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:35.575918 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:35.576413 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 02:31:35.576789 (+ 376us) client_negotiation.cc:240] Negotiation successful
0620 02:31:35.577442 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":359,"threads_started":1}
I20260620 02:31:35.580564 13113 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260620 02:31:35.580905 13113 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:35.581713 13113 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:35.583433 12757 ts_manager.cc:194] Registered new tserver with Master: 8bb70000273943cba13ac7c016f1fcc8 (127.0.97.2:37421)
I20260620 02:31:35.584251 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34585
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:31:35.624168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:31:35.816279 12970 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
W20260620 02:31:35.991755 13119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:35.992338 13119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:35.992491 13119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:36.000501 13119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:36.000687 13119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:36.000860 13119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:31:36.012425 13119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:57769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:36.013963 13119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:36.015374 13119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:36.031980 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:36.032071 13127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:36.032020 13124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:36.040472 13119 server_base.cc:1034] running on GCE node
I20260620 02:31:36.041358 13119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:36.042546 13119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:36.043849 13119 hybrid_clock.cc:648] HybridClock initialized: now 1781922696043790 us; error 48 us; skew 500 ppm
I20260620 02:31:36.044240 13119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:36 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:36.047961 13119 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:31:36.050047 13119 webserver.cc:460] Webserver started at http://127.0.97.3:34327/ using document root <none> and password file <none>
I20260620 02:31:36.050926 13119 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:36.051113 13119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:36.051503 13119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:31:36 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:36.068135 13119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "fd9c1c65bcca426c939270d7b05e3aa2"
format_stamp: "Formatted at 2026-06-20 02:31:36 on dist-test-slave-r706"
server_key: "dc1226a0ff4acb8ab8d541b93c9521a7"
server_key_iv: "b4980a83c0be43d3068c7d6106835970"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:36.069638 13119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fd9c1c65bcca426c939270d7b05e3aa2"
format_stamp: "Formatted at 2026-06-20 02:31:36 on dist-test-slave-r706"
server_key: "dc1226a0ff4acb8ab8d541b93c9521a7"
server_key_iv: "b4980a83c0be43d3068c7d6106835970"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:36.076602 13119 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:31:36.093060 13134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:36.094754 13119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:31:36.094969 13119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "fd9c1c65bcca426c939270d7b05e3aa2"
format_stamp: "Formatted at 2026-06-20 02:31:36 on dist-test-slave-r706"
server_key: "dc1226a0ff4acb8ab8d541b93c9521a7"
server_key_iv: "b4980a83c0be43d3068c7d6106835970"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:36.095218 13119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:36.113574 13119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:36.118696 13119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:36.119120 13119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:36.120371 13119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:36.122335 13119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:36.122437 13119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:36.122645 13119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:36.122768 13119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:36.164911 13119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43107
I20260620 02:31:36.165130 13246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43107 every 8 connection(s)
I20260620 02:31:36.166544 13119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:31:36.172247 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13119
I20260620 02:31:36.172462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:31:36.173076 388 external_mini_cluster.cc:1428] Setting key f6380c8ad560e1a092ff6b9316bf0b8d
Jun 20 02:31:36 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:36.190651 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:36.170040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51573 (local address 127.0.97.62:35481)
0620 02:31:36.170739 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:36.170746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:36.171864 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 02:31:36.174503 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:36.174509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:36.174603 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:36.174726 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:36.176730 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.177670 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.179545 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.180112 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.180290 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:36.184595 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:36.184600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:36.184604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:36.184650 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:36.187317 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.188184 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.188196 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.188199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.188325 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.188792 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.188799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.188802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.189161 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:36.189320 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:36.190039 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 02:31:36.190278 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":360,"threads_started":1}
I20260620 02:31:36.191689 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:36.170759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.3:51573)
0620 02:31:36.171661 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:36.171705 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:36.174160 (+ 2455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:36.175057 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:36.175067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:36.175711 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:36.176552 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:36.176572 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.177852 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.177856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:36.179376 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:36.179387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.180429 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.180446 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:36.180737 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:36.181613 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:36.181637 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:36.184311 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:36.187492 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:36.187501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:36.187526 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:36.188051 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:36.188448 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:36.188453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:36.188457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:36.188673 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:36.189302 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:36.189310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:36.189659 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 02:31:36.189945 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 02:31:36.190519 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":407,"threads_started":1}
I20260620 02:31:36.193107 13247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260620 02:31:36.193425 13247 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:36.194202 13247 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:36.195848 12757 ts_manager.cc:194] Registered new tserver with Master: fd9c1c65bcca426c939270d7b05e3aa2 (127.0.97.3:43107)
I20260620 02:31:36.196777 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51573
I20260620 02:31:36.201606 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:31:36.203190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 12837
Jun 20 02:31:36 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:31:36.220160 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:36.204159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.0.97.1:33665)
0620 02:31:36.204845 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:36.204852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:36.204877 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:31:36.204939 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:36.204944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:36.205195 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:36.205330 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:36.206773 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.207983 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.209474 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.210051 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.210149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:36.214522 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:36.214533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:36.214546 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:36.214578 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:36.217618 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.218290 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.218296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.218298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.218441 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.218849 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.218852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.218854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.219163 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:36.219384 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:36.219667 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 02:31:36.219879 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":387,"threads_started":1}
I20260620 02:31:36.221544 12965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:36.306555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 12837
I20260620 02:31:36.322844 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 12976
Jun 20 02:31:36 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:31:36.340463 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:36.323872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.0.97.2:37421)
0620 02:31:36.324493 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:36.324500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:36.324532 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:31:36.324616 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:36.324622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:36.324851 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:36.325084 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:36.326493 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.327552 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.329170 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.329855 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.329977 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:36.334596 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:36.334609 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:36.334625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:36.334671 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:36.337739 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.338438 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.338446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.338448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.338591 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.338972 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.338976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.338978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.339383 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:36.339619 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:36.339966 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 02:31:36.340169 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":350,"threads_started":1}
I20260620 02:31:36.341826 13108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:36.424623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 12976
I20260620 02:31:36.445214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13119
Jun 20 02:31:36 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:31:36.463662 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:36.446345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.0.97.3:43107)
0620 02:31:36.447007 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:36.447014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:36.447044 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:31:36.447107 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:36.447113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:36.447328 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:36.447527 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:36.448921 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.450107 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.451501 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:36.452088 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:36.452240 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:36.457262 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:36.457277 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:36.457293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:36.457336 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:36.460387 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.461353 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.461367 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.461369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.461522 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:36.461980 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:36.461984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:36.461986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:36.462400 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:36.462660 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:36.462985 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 02:31:36.463206 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":346,"threads_started":1}
I20260620 02:31:36.465253 13242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:36.551501 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13119
I20260620 02:31:36.569350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 12725
I20260620 02:31:36.570713 12787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:36.679734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 12725
I20260620 02:31:36.705118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--webserver_interface=127.0.97.62
--webserver_port=36335
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35481
--ranger_config_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:37.049065 13259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:37.049400 13259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:37.049566 13259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:37.057521 13259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:37.057646 13259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:37.057729 13259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:37.057834 13259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:37.057927 13259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:37.068367 13259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35481
--ranger_config_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:37.069800 13259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:37.071208 13259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:37.084254 13265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:37.084232 13267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:37.084379 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:37.085088 13259 server_base.cc:1034] running on GCE node
I20260620 02:31:37.086017 13259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:37.087750 13259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:37.089058 13259 hybrid_clock.cc:648] HybridClock initialized: now 1781922697089001 us; error 71 us; skew 500 ppm
I20260620 02:31:37.089475 13259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:37 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:37.093186 13259 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:37.094969 13259 webserver.cc:460] Webserver started at http://127.0.97.62:36335/ using document root <none> and password file <none>
I20260620 02:31:37.095731 13259 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:37.095911 13259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:31:37 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:37.184140 13259 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:31:37.187553 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:37.189139 13259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:31:37.189353 13259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "07ae14bf2ce042de826aa8befddc5514"
format_stamp: "Formatted at 2026-06-20 02:31:32 on dist-test-slave-r706"
server_key: "77f4ebb5c5795c15ffb5d82aae5612f3"
server_key_iv: "ac511ebe94f2d87f95fd9e3f4f849b95"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:37.190433 13259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:37.215101 13259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:37.219718 13259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:37.220062 13259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:37.240165 13259 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35481
I20260620 02:31:37.240195 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35481 every 8 connection(s)
I20260620 02:31:37.241793 13259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:37.242728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13259
I20260620 02:31:37.243935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33665
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35481
--webserver_port=44403
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:31:37.252818 13327 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:31:37.256563 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Bootstrap starting.
I20260620 02:31:37.276396 13327 log.cc:824] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:37.286981 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:31:37.287654 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Bootstrap complete.
I20260620 02:31:37.298224 13327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:37.299244 13327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae14bf2ce042de826aa8befddc5514, State: Initialized, Role: FOLLOWER
I20260620 02:31:37.300101 13327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:37.300388 13327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:31:37.300552 13327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:31:37.300769 13327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:31:37.305326 13327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:37.305903 13327 leader_election.cc:302] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07ae14bf2ce042de826aa8befddc5514; no voters:
I20260620 02:31:37.306684 13327 leader_election.cc:288] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:31:37.306830 13331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:31:37.308027 13331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [term 2 LEADER]: Becoming Leader. State: Replica: 07ae14bf2ce042de826aa8befddc5514, State: Running, Role: LEADER
I20260620 02:31:37.308760 13331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260620 02:31:37.309190 13327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:37.311679 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07ae14bf2ce042de826aa8befddc5514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260620 02:31:37.312182 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:37.312230 13333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07ae14bf2ce042de826aa8befddc5514. Latest consensus state: current_term: 2 leader_uuid: "07ae14bf2ce042de826aa8befddc5514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ae14bf2ce042de826aa8befddc5514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260620 02:31:37.312580 13333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:37.316675 13337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:37.322773 13337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:37.324163 13337 catalog_manager.cc:1259] Loaded cluster ID: fce6d92c00bf4e62afd38d1d6c7899a6
I20260620 02:31:37.324319 13337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:31:37.328615 13337 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:31:37.330282 13337 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Loaded TSK: 0
I20260620 02:31:37.331988 13337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:31:37.339249 13327 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 02:31:37.694657 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:37.695192 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:37.695379 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:37.707295 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:37.707557 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:37.707789 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:31:37.724381 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44403
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:37.726312 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:37.728209 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:37.751085 13364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:37.753492 13365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:37.757131 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:37.757750 13329 server_base.cc:1034] running on GCE node
I20260620 02:31:37.758697 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:37.760177 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:37.761513 13329 hybrid_clock.cc:648] HybridClock initialized: now 1781922697761430 us; error 79 us; skew 500 ppm
I20260620 02:31:37.761981 13329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:37 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:37.767096 13329 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:31:37.770704 13329 webserver.cc:460] Webserver started at http://127.0.97.1:44403/ using document root <none> and password file <none>
I20260620 02:31:37.771875 13329 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:37.772073 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:31:37 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:37.811677 13329 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
I20260620 02:31:37.819932 13375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:37.822162 13329 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260620 02:31:37.822496 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "331b6d7ce97344ad9751d951196b4e54"
format_stamp: "Formatted at 2026-06-20 02:31:34 on dist-test-slave-r706"
server_key: "0aa0b9a240d3e693db6160c60137f799"
server_key_iv: "de37a807da5c4e9f3791bb9be3a392f0"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:37.824088 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:37.889665 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:37.895525 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:37.895928 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:37.903306 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:37.911559 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:37.911761 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:31:37.911938 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:37.912067 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:38.011776 13329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33665
I20260620 02:31:38.011822 13488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33665 every 8 connection(s)
I20260620 02:31:38.013396 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:31:38.021843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13329
I20260620 02:31:38.022708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37421
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35481
--webserver_port=35885
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:31:38 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:38.048817 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:38.017282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58417 (local address 127.0.97.62:35481)
0620 02:31:38.018098 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:38.018112 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:38.018865 (+ 753us) server_negotiation.cc:407] Connection header received
0620 02:31:38.021236 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:38.021257 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:38.021737 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:38.022120 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:38.024512 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.026559 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.028185 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.028825 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.029071 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:38.037479 (+ 8408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:38.037496 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:38.037519 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:38.037576 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:38.042415 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:38.043656 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:38.043668 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:38.043677 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:38.043914 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:38.044535 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:38.044542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:38.044545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:38.045379 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:38.045628 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:38.046739 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0620 02:31:38.047210 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":341,"threads_started":1}
I20260620 02:31:38.050087 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:38.017427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.1:58417)
0620 02:31:38.018655 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:38.018705 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:38.020564 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:38.022857 (+ 2293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:38.022868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:38.023515 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:38.024236 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:38.024262 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.026796 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.026799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:38.028024 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:38.028040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.029098 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.029107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:38.029374 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:38.030198 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:38.030227 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:38.037038 (+ 6811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:38.042636 (+ 5598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:38.042649 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:38.042689 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:38.043444 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:38.044089 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:38.044097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:38.044104 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:38.044378 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:38.045590 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:38.045602 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:38.046087 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 02:31:38.046467 (+ 380us) client_negotiation.cc:240] Negotiation successful
0620 02:31:38.047167 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":697,"threads_started":1}
I20260620 02:31:38.054258 13489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260620 02:31:38.054672 13489 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:38.055809 13489 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
W20260620 02:31:38.059707 13489 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35481 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:31:38.059962 13489 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:38.060273 13489 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:38.061856 13489 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:38.062158 13489 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
W20260620 02:31:38.063635 13489 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:31:38.453315 13493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:38.453822 13493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:38.454082 13493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:38.463775 13493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:38.463937 13493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:38.464135 13493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:31:38.474860 13493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35885
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:38.476477 13493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:38.477912 13493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:38.494256 13507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:38.494256 13508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:38.494673 13510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:38.495414 13493 server_base.cc:1034] running on GCE node
I20260620 02:31:38.496294 13493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:38.504230 13493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:38.505513 13493 hybrid_clock.cc:648] HybridClock initialized: now 1781922698505441 us; error 68 us; skew 500 ppm
I20260620 02:31:38.505870 13493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:38 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:38.509410 13493 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:31:38.511102 13493 webserver.cc:460] Webserver started at http://127.0.97.2:35885/ using document root <none> and password file <none>
I20260620 02:31:38.511796 13493 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:38.511940 13493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:31:38 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:38.554812 13493 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.005s sys 0.000s
I20260620 02:31:38.561476 13518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:38.563547 13493 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 02:31:38.563760 13493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "8bb70000273943cba13ac7c016f1fcc8"
format_stamp: "Formatted at 2026-06-20 02:31:35 on dist-test-slave-r706"
server_key: "e2f1506e4adf7d909f7960f098900472"
server_key_iv: "79b1c99ad56e9476f8fe5650b51a6d30"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:38.565274 13493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:38.605933 13493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:38.611778 13493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:38.612175 13493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:38.613529 13493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:38.615438 13493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:38.615615 13493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:38.615777 13493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:38.615888 13493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:31:38 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781922698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:38.694293 13493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37421
I20260620 02:31:38.694350 13632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37421 every 8 connection(s)
I20260620 02:31:38.696259 13493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:31:38.701200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13493
I20260620 02:31:38.702167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43107
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35481
--webserver_port=34327
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:31:38 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:38.729375 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:38.699761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.2:39325)
0620 02:31:38.702552 (+ 2791us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:38.702612 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:38.707729 (+ 5117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:38.708861 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:38.708873 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:38.709609 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:38.710560 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:38.710589 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.712240 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.712246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:38.714363 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:38.714380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.715329 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.715338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:38.715646 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:38.716523 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:38.716547 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:38.719003 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:38.723780 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:38.723793 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:38.723815 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:38.724504 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:38.725075 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:38.725083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:38.725090 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:38.725356 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:38.726079 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:38.726091 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:38.726590 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 02:31:38.726903 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 02:31:38.727610 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2452,"thread_start_us":5407,"threads_started":1}
I20260620 02:31:38.731184 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:38.700436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39325 (local address 127.0.97.62:35481)
0620 02:31:38.701589 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:38.701596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:38.705076 (+ 3480us) server_negotiation.cc:407] Connection header received
0620 02:31:38.708089 (+ 3013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:38.708096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:38.708195 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:38.708282 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:38.710801 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.712003 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.714515 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:38.715043 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:38.715208 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:38.719267 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:38.719272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:38.719275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:38.719311 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:38.723583 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:38.724691 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:38.724696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:38.724700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:38.724888 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:38.725513 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:38.725520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:38.725526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:38.725922 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:38.730249 (+ 4327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:38.730410 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:31:38.730701 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":5743,"threads_started":1}
I20260620 02:31:38.733357 13633 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260620 02:31:38.733716 13633 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:38.734601 13633 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:38.736693 13292 ts_manager.cc:194] Registered new tserver with Master: 8bb70000273943cba13ac7c016f1fcc8 (127.0.97.2:37421)
I20260620 02:31:38.739002 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39325
I20260620 02:31:39.063794 13489 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:39.064270 13489 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:39.065840 13292 ts_manager.cc:194] Registered new tserver with Master: 331b6d7ce97344ad9751d951196b4e54 (127.0.97.1:33665)
I20260620 02:31:39.067238 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58417
Jun 20 02:31:39 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:31:39.165145 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:39.165491 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:39.165673 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:39.173965 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:39.174167 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:39.174365 13637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:31:39.184875 13637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34327
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:39.186403 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:39.187781 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:39.205652 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:39.205731 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:39.206929 13637 server_base.cc:1034] running on GCE node
W20260620 02:31:39.207353 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:39.208143 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:39.209374 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:39.210625 13637 hybrid_clock.cc:648] HybridClock initialized: now 1781922699210552 us; error 57 us; skew 500 ppm
I20260620 02:31:39.210959 13637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:39 dist-test-slave-r706 krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:39.214429 13637 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:31:39.216326 13637 webserver.cc:460] Webserver started at http://127.0.97.3:34327/ using document root <none> and password file <none>
I20260620 02:31:39.217139 13637 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:39.217314 13637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:31:39 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:39.253978 13637 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.000s sys 0.004s
I20260620 02:31:39.257486 13654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:39.259155 13637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:31:39.259367 13637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "fd9c1c65bcca426c939270d7b05e3aa2"
format_stamp: "Formatted at 2026-06-20 02:31:36 on dist-test-slave-r706"
server_key: "dc1226a0ff4acb8ab8d541b93c9521a7"
server_key_iv: "b4980a83c0be43d3068c7d6106835970"
server_key_version: "kuduclusterkey@0"
I20260620 02:31:39.260447 13637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:39.283191 13637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:39.287870 13637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:39.288183 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:39.289312 13637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:39.290870 13637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:39.290995 13637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:39.291136 13637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:39.291231 13637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:39.348771 13637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43107
I20260620 02:31:39.348815 13766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43107 every 8 connection(s)
I20260620 02:31:39.350298 13637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:31:39.358596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13637
Jun 20 02:31:39 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:39.415110 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.359725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.0.97.62:35481)
0620 02:31:39.391512 (+ 31787us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.391522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.391555 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:31:39.391669 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.391678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.391815 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.392004 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:39.396002 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.397172 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.399478 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.400657 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.400801 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.404865 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:39.404872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:39.404876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:39.404925 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:39.409163 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:39.411386 (+ 2223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:39.411391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:39.411395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:39.411547 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:39.412401 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:39.412405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:39.412408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:39.412769 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:39.413053 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.414464 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.414707 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31633,"thread_start_us":29914,"threads_started":1}
I20260620 02:31:39.421943 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.355190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53479 (local address 127.0.97.62:35481)
0620 02:31:39.390444 (+ 35254us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.390452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.395905 (+ 5453us) server_negotiation.cc:407] Connection header received
0620 02:31:39.400061 (+ 4156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.400066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.400162 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.400270 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:39.402622 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.403951 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.406654 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.407458 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.407571 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.412715 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:39.412723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:39.412728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:39.412780 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:39.417247 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:39.418495 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:39.418501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:39.418505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:39.418655 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:39.419532 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:39.419537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:39.419540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:39.419908 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:39.420054 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.421266 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.421538 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35097,"thread_start_us":35992,"threads_started":1}
I20260620 02:31:39.423830 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.354435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.3:53479)
0620 02:31:39.395701 (+ 41266us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.395773 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.399716 (+ 3943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.400628 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.400641 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.401379 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:39.402343 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.402377 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.404197 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.404202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.406471 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.406489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.407987 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.407995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.408324 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.409454 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:39.409486 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:39.412385 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:39.417444 (+ 5059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:39.417457 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:39.417478 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:39.418345 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:39.419104 (+ 759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:39.419110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:39.419113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:39.419381 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:39.420200 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:39.420208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:39.420679 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.421738 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.422372 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40854,"thread_start_us":40573,"threads_started":1}
I20260620 02:31:39.425630 13767 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260620 02:31:39.425981 13767 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:39.426851 13767 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260620 02:31:39.428539 13292 ts_manager.cc:194] Registered new tserver with Master: fd9c1c65bcca426c939270d7b05e3aa2 (127.0.97.3:43107)
Jun 20 02:31:39 dist-test-slave-r706 krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:31:39.429891 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53479
I20260620 02:31:39.449220 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:31:39.490624 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.471164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.0.97.62:35481)
0620 02:31:39.471374 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.471382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.473617 (+ 2235us) server_negotiation.cc:407] Connection header received
0620 02:31:39.473975 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.473981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.474080 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.474182 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:39.476248 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.477221 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.478997 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.479503 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.479643 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.483960 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:39.483968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:39.483974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:39.484024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:39.487032 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:39.487983 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:39.487990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:39.487994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:39.488199 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:39.488695 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:39.488700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:39.488703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:39.489165 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:39.489421 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.490000 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.490274 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:31:39.498628 13292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:31:39.510439 13292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:31:39.575712 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.540013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.0.97.2:37421)
0620 02:31:39.560336 (+ 20323us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.560348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.563297 (+ 2949us) server_negotiation.cc:407] Connection header received
0620 02:31:39.563545 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.563555 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.563913 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.564154 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.566723 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.568329 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.571766 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.573602 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.573754 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.574143 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.575059 (+ 916us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.575336 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20102,"thread_start_us":20346,"threads_started":1}
I20260620 02:31:39.576442 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.535368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33665 (local address 127.0.0.1:41898)
0620 02:31:39.545925 (+ 10557us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.545955 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.546150 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.565063 (+ 18913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.565070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.565121 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.565539 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.565551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.569378 (+ 3827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.569383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.571531 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.571542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.575474 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.575482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.575657 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.575696 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.575776 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.576000 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10383,"mutex_wait_us":6,"thread_start_us":27397,"threads_started":1}
I20260620 02:31:39.576876 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.539656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37421 (local address 127.0.0.1:42406)
0620 02:31:39.563120 (+ 23464us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.563149 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.563361 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.566120 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.566125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.566153 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.566503 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.566515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.568547 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.568552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.571485 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.571513 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.574393 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.574400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.574663 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.574750 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.576308 (+ 1558us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.576475 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23290,"mutex_wait_us":71,"thread_start_us":5952,"threads_started":1}
I20260620 02:31:39.580961 13568 tablet_service.cc:1467] Processing CreateTablet for tablet 8022b984d612453698f327c5303c0538 (DEFAULT_TABLE table=test-table [id=a1b033fb49104ccd8be68387dc5b5e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:39.582618 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8022b984d612453698f327c5303c0538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:39.587800 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.535680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.0.97.1:33665)
0620 02:31:39.564096 (+ 28416us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.564106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.564134 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:31:39.564230 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.564239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.564458 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.564972 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.567666 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.569204 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.571796 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.574536 (+ 2740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.576938 (+ 2402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.587106 (+ 10168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.587212 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.587429 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28214,"thread_start_us":28987,"threads_started":1}
I20260620 02:31:39.593668 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 8022b984d612453698f327c5303c0538 (DEFAULT_TABLE table=test-table [id=a1b033fb49104ccd8be68387dc5b5e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:39.595482 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8022b984d612453698f327c5303c0538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:39.600230 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.531840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43107 (local address 127.0.0.1:54532)
0620 02:31:39.537638 (+ 5798us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.537689 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.537881 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.592762 (+ 54881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.592769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.592812 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.593195 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.593206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.595211 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.595217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.597428 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.597453 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.599405 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.599412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.599595 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.599634 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.599709 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.599850 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5495,"thread_start_us":5362,"threads_started":1}
I20260620 02:31:39.601488 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.532848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.0.97.3:43107)
0620 02:31:39.592101 (+ 59253us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.592110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.592146 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:31:39.592250 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.592259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.592512 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.592747 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.593413 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.594972 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.597672 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.599156 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.600621 (+ 1465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.600860 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.600934 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.601153 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57607,"thread_start_us":56170,"threads_started":1}
I20260620 02:31:39.605813 13702 tablet_service.cc:1467] Processing CreateTablet for tablet 8022b984d612453698f327c5303c0538 (DEFAULT_TABLE table=test-table [id=a1b033fb49104ccd8be68387dc5b5e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:39.607576 13702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8022b984d612453698f327c5303c0538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:39.631232 13789 tablet_bootstrap.cc:492] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Bootstrap starting.
I20260620 02:31:39.635351 13789 tablet_bootstrap.cc:654] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:39.636741 13789 log.cc:824] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:39.647109 13789 tablet_bootstrap.cc:492] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: No bootstrap required, opened a new log
I20260620 02:31:39.647509 13789 ts_tablet_manager.cc:1397] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260620 02:31:39.657876 13791 tablet_bootstrap.cc:492] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Bootstrap starting.
I20260620 02:31:39.662902 13791 tablet_bootstrap.cc:654] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:39.664628 13792 tablet_bootstrap.cc:492] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Bootstrap starting.
I20260620 02:31:39.666133 13791 log.cc:824] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:39.669970 13792 tablet_bootstrap.cc:654] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:39.671612 13792 log.cc:824] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:39.671576 13789 raft_consensus.cc:348] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.672669 13789 raft_consensus.cc:374] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:39.673081 13789 raft_consensus.cc:729] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb70000273943cba13ac7c016f1fcc8, State: Initialized, Role: FOLLOWER
I20260620 02:31:39.674495 13789 consensus_queue.cc:260] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.687479 13791 tablet_bootstrap.cc:492] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: No bootstrap required, opened a new log
I20260620 02:31:39.687847 13791 ts_tablet_manager.cc:1397] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.003s
I20260620 02:31:39.699039 13791 raft_consensus.cc:348] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.699563 13791 raft_consensus.cc:374] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:39.699692 13791 raft_consensus.cc:729] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9c1c65bcca426c939270d7b05e3aa2, State: Initialized, Role: FOLLOWER
I20260620 02:31:39.700368 13791 consensus_queue.cc:260] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.701642 13633 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
I20260620 02:31:39.702327 13789 ts_tablet_manager.cc:1428] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Time spent starting tablet: real 0.055s user 0.021s sys 0.020s
I20260620 02:31:39.704828 13792 tablet_bootstrap.cc:492] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: No bootstrap required, opened a new log
I20260620 02:31:39.705264 13792 ts_tablet_manager.cc:1397] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.013s
W20260620 02:31:39.708878 13634 tablet.cc:2369] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:39.714545 13791 ts_tablet_manager.cc:1428] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Time spent starting tablet: real 0.027s user 0.005s sys 0.011s
I20260620 02:31:39.715128 13767 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
I20260620 02:31:39.716974 13792 raft_consensus.cc:348] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.717577 13792 raft_consensus.cc:374] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:39.717739 13792 raft_consensus.cc:729] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331b6d7ce97344ad9751d951196b4e54, State: Initialized, Role: FOLLOWER
I20260620 02:31:39.718482 13792 consensus_queue.cc:260] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.738217 13489 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
I20260620 02:31:39.739166 13792 ts_tablet_manager.cc:1428] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Time spent starting tablet: real 0.034s user 0.013s sys 0.019s
W20260620 02:31:39.767241 13490 tablet.cc:2369] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:31:39.858397 13768 tablet.cc:2369] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:39.926434 13797 raft_consensus.cc:482] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:31:39.926782 13797 raft_consensus.cc:504] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.928555 13797 leader_election.cc:288] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd9c1c65bcca426c939270d7b05e3aa2 (127.0.97.3:43107), 8bb70000273943cba13ac7c016f1fcc8 (127.0.97.2:37421)
I20260620 02:31:39.933729 13795 raft_consensus.cc:482] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:31:39.934057 13795 raft_consensus.cc:504] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.935990 13795 leader_election.cc:288] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd9c1c65bcca426c939270d7b05e3aa2 (127.0.97.3:43107), 331b6d7ce97344ad9751d951196b4e54 (127.0.97.1:33665)
I20260620 02:31:39.936861 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.928969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59849 (local address 127.0.97.3:43107)
0620 02:31:39.929165 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.929170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.929729 (+ 559us) server_negotiation.cc:407] Connection header received
0620 02:31:39.930025 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.930031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.930116 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.930230 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.931084 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.932027 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.934436 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.935689 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.935835 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.935947 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.936441 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.936600 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:31:39.936906 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.928852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43107 (local address 127.0.97.1:59849)
0620 02:31:39.929570 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.929592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.929776 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.930417 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.930429 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.930480 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.930923 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.930939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.932159 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.932164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.934267 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.934280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.935928 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.935936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.936123 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.936166 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.936278 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.936436 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"spinlock_wait_cycles":22272,"thread_start_us":360,"threads_started":1}
I20260620 02:31:39.937100 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.930076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46403 (local address 127.0.97.2:37421)
0620 02:31:39.930315 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.930321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.930580 (+ 259us) server_negotiation.cc:407] Connection header received
0620 02:31:39.930764 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.930769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.930850 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.930955 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.931569 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.932366 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.934657 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.936006 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.936172 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.936281 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.936633 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.936782 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:31:39.937846 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.929712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37421 (local address 127.0.97.1:46403)
0620 02:31:39.930440 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.930454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.930616 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.930980 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.930983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.931026 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.931416 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.931427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.932542 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.932550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.934414 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.934429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.936251 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.936257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.936444 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.936476 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.937247 (+ 771us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.937388 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"spinlock_wait_cycles":6272,"thread_start_us":418,"threads_started":1}
I20260620 02:31:39.938795 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8022b984d612453698f327c5303c0538" candidate_uuid: "331b6d7ce97344ad9751d951196b4e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb70000273943cba13ac7c016f1fcc8" is_pre_election: true
I20260620 02:31:39.939356 13588 raft_consensus.cc:2454] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 331b6d7ce97344ad9751d951196b4e54 in term 0.
I20260620 02:31:39.940306 13722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8022b984d612453698f327c5303c0538" candidate_uuid: "331b6d7ce97344ad9751d951196b4e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" is_pre_election: true
I20260620 02:31:39.940821 13722 raft_consensus.cc:2454] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 331b6d7ce97344ad9751d951196b4e54 in term 0.
I20260620 02:31:39.941715 13376 leader_election.cc:302] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 331b6d7ce97344ad9751d951196b4e54, 8bb70000273943cba13ac7c016f1fcc8; no voters:
I20260620 02:31:39.942512 13797 raft_consensus.cc:2781] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:31:39.942677 13797 raft_consensus.cc:482] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:31:39.942791 13797 raft_consensus.cc:3037] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:39.946846 13797 raft_consensus.cc:504] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.947358 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.937371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44805 (local address 127.0.97.3:43107)
0620 02:31:39.939018 (+ 1647us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.939024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.939856 (+ 832us) server_negotiation.cc:407] Connection header received
0620 02:31:39.940104 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.940108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.940189 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.940249 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.941149 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.942247 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.944766 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.946068 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.946166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.946291 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.946928 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.947067 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1550,"threads_started":1}
I20260620 02:31:39.947969 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.937200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43107 (local address 127.0.97.2:44805)
0620 02:31:39.939763 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.939783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.939956 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.940442 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.940446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.940471 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.940932 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.940948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.942425 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.942429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.944557 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.944573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.946500 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.946507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.946711 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.946753 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.947503 (+ 750us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.947641 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2413,"threads_started":1}
I20260620 02:31:39.948239 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.937263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55561 (local address 127.0.97.1:33665)
0620 02:31:39.937490 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:39.937503 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:39.938101 (+ 598us) server_negotiation.cc:407] Connection header received
0620 02:31:39.938398 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:39.938403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:39.938478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:39.938543 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:39.939368 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.940755 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.944089 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.945977 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.946126 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.946271 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:39.947771 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0620 02:31:39.947967 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:31:39.948678 13797 leader_election.cc:288] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [CANDIDATE]: Term 1 election: Requested vote from peers fd9c1c65bcca426c939270d7b05e3aa2 (127.0.97.3:43107), 8bb70000273943cba13ac7c016f1fcc8 (127.0.97.2:37421)
I20260620 02:31:39.948696 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:39.936477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33665 (local address 127.0.97.2:55561)
0620 02:31:39.937987 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:39.938010 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:39.938233 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:39.938750 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:39.938757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:39.938789 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:39.939193 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.939209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.940973 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.940985 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.943844 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:39.943866 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:39.946843 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:39.946853 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:39.947092 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:39.947123 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:31:39.947231 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:31:39.947426 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"thread_start_us":1184,"threads_started":1}
I20260620 02:31:39.948647 13722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8022b984d612453698f327c5303c0538" candidate_uuid: "331b6d7ce97344ad9751d951196b4e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9c1c65bcca426c939270d7b05e3aa2"
I20260620 02:31:39.949250 13722 raft_consensus.cc:3037] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:39.949352 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8022b984d612453698f327c5303c0538" candidate_uuid: "331b6d7ce97344ad9751d951196b4e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb70000273943cba13ac7c016f1fcc8"
I20260620 02:31:39.949586 13588 raft_consensus.cc:3037] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:39.949734 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8022b984d612453698f327c5303c0538" candidate_uuid: "8bb70000273943cba13ac7c016f1fcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "331b6d7ce97344ad9751d951196b4e54" is_pre_election: true
I20260620 02:31:39.950223 13444 raft_consensus.cc:2379] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8bb70000273943cba13ac7c016f1fcc8 in current term 1: Already voted for candidate 331b6d7ce97344ad9751d951196b4e54 in this term.
I20260620 02:31:39.951112 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8022b984d612453698f327c5303c0538" candidate_uuid: "8bb70000273943cba13ac7c016f1fcc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" is_pre_election: true
I20260620 02:31:39.951970 13521 leader_election.cc:302] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8bb70000273943cba13ac7c016f1fcc8; no voters: 331b6d7ce97344ad9751d951196b4e54, fd9c1c65bcca426c939270d7b05e3aa2
I20260620 02:31:39.952836 13722 raft_consensus.cc:2454] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331b6d7ce97344ad9751d951196b4e54 in term 1.
I20260620 02:31:39.953251 13376 leader_election.cc:302] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 331b6d7ce97344ad9751d951196b4e54, fd9c1c65bcca426c939270d7b05e3aa2; no voters:
I20260620 02:31:39.953558 13797 raft_consensus.cc:2781] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:39.953687 13588 raft_consensus.cc:2454] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331b6d7ce97344ad9751d951196b4e54 in term 1.
I20260620 02:31:39.953858 13795 raft_consensus.cc:2726] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 02:31:39.954169 13797 raft_consensus.cc:686] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 1 LEADER]: Becoming Leader. State: Replica: 331b6d7ce97344ad9751d951196b4e54, State: Running, Role: LEADER
I20260620 02:31:39.954703 13797 consensus_queue.cc:237] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } }
I20260620 02:31:39.959543 13292 catalog_manager.cc:5495] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 331b6d7ce97344ad9751d951196b4e54 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "331b6d7ce97344ad9751d951196b4e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "331b6d7ce97344ad9751d951196b4e54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 } health_report { overall_health: UNKNOWN } } }
I20260620 02:31:40.051429 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:40.045408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.97.1:33665)
0620 02:31:40.045667 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:40.045674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:40.045878 (+ 204us) server_negotiation.cc:407] Connection header received
0620 02:31:40.046141 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:40.046145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:40.046256 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:40.046434 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:31:40.046831 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.047568 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.049072 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.049502 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.049661 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:40.050247 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:31:40.050795 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:31:40.050956 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:40.051037 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 02:31:40.051125 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:31:40.060426 13588 raft_consensus.cc:1261] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 1 FOLLOWER]: Refusing update from remote peer 331b6d7ce97344ad9751d951196b4e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:40.060415 13722 raft_consensus.cc:1261] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 1 FOLLOWER]: Refusing update from remote peer 331b6d7ce97344ad9751d951196b4e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:40.061602 13797 consensus_queue.cc:1035] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bb70000273943cba13ac7c016f1fcc8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:31:40.062012 13810 consensus_queue.cc:1035] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9c1c65bcca426c939270d7b05e3aa2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:31:40.078291 13815 mvcc.cc:205] Tried to move back new op lower bound from 7298755379435589632 to 7298755379018616832. Current Snapshot: MvccSnapshot[applied={T|T < 7298755379435589632}]
I20260620 02:31:40.092231 13291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:31:40.092556 13291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:31:40.096792 13291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ae14bf2ce042de826aa8befddc5514: Sending DeleteTablet for 3 replicas of tablet 8022b984d612453698f327c5303c0538
I20260620 02:31:40.098264 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8022b984d612453698f327c5303c0538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41898
I20260620 02:31:40.098492 13702 tablet_service.cc:1514] Processing DeleteTablet for tablet 8022b984d612453698f327c5303c0538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54532
I20260620 02:31:40.098688 13568 tablet_service.cc:1514] Processing DeleteTablet for tablet 8022b984d612453698f327c5303c0538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42406
I20260620 02:31:40.100062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13329
I20260620 02:31:40.118942 13824 tablet_replica.cc:330] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: stopping tablet replica
I20260620 02:31:40.119565 13824 raft_consensus.cc:2229] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:31:40.120378 13824 raft_consensus.cc:2258] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:40.122227 13826 tablet_replica.cc:330] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: stopping tablet replica
I20260620 02:31:40.122727 13826 raft_consensus.cc:2229] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:31:40.123291 13826 raft_consensus.cc:2258] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:40.123879 13824 ts_tablet_manager.cc:1905] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:40.125150 13825 tablet_replica.cc:330] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: stopping tablet replica
I20260620 02:31:40.125643 13825 raft_consensus.cc:2229] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:31:40.126956 13825 raft_consensus.cc:2258] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:40.127995 13826 ts_tablet_manager.cc:1905] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:40.130824 13824 ts_tablet_manager.cc:1918] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:40.131038 13824 log.cc:1196] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/8022b984d612453698f327c5303c0538
I20260620 02:31:40.131626 13824 ts_tablet_manager.cc:1939] T 8022b984d612453698f327c5303c0538 P 331b6d7ce97344ad9751d951196b4e54: Deleting consensus metadata
I20260620 02:31:40.133683 13279 catalog_manager.cc:4841] TS 331b6d7ce97344ad9751d951196b4e54 (127.0.97.1:33665): tablet 8022b984d612453698f327c5303c0538 (table test-table [id=a1b033fb49104ccd8be68387dc5b5e1d]) successfully deleted
I20260620 02:31:40.134140 13825 ts_tablet_manager.cc:1905] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:40.137555 13826 ts_tablet_manager.cc:1918] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:40.137760 13826 log.cc:1196] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/8022b984d612453698f327c5303c0538
I20260620 02:31:40.138376 13826 ts_tablet_manager.cc:1939] T 8022b984d612453698f327c5303c0538 P 8bb70000273943cba13ac7c016f1fcc8: Deleting consensus metadata
I20260620 02:31:40.140475 13276 catalog_manager.cc:4841] TS 8bb70000273943cba13ac7c016f1fcc8 (127.0.97.2:37421): tablet 8022b984d612453698f327c5303c0538 (table test-table [id=a1b033fb49104ccd8be68387dc5b5e1d]) successfully deleted
I20260620 02:31:40.140580 13825 ts_tablet_manager.cc:1918] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:40.140766 13825 log.cc:1196] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/8022b984d612453698f327c5303c0538
I20260620 02:31:40.141635 13825 ts_tablet_manager.cc:1939] T 8022b984d612453698f327c5303c0538 P fd9c1c65bcca426c939270d7b05e3aa2: Deleting consensus metadata
I20260620 02:31:40.143905 13276 catalog_manager.cc:4841] TS fd9c1c65bcca426c939270d7b05e3aa2 (127.0.97.3:43107): tablet 8022b984d612453698f327c5303c0538 (table test-table [id=a1b033fb49104ccd8be68387dc5b5e1d]) successfully deleted
I20260620 02:31:40.150244 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:40.104341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.97.1:33665)
0620 02:31:40.104556 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:40.104566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:40.121568 (+ 17002us) server_negotiation.cc:407] Connection header received
0620 02:31:40.123975 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:40.123982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:40.124072 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:40.124185 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:40.126182 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.127472 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.132109 (+ 4637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.132848 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.132986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:40.141694 (+ 8708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:40.141714 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:40.141734 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:40.141787 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:40.146462 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:40.147380 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:40.147393 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:40.147397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:40.147572 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:40.148101 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:40.148107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:40.148110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:40.148681 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:40.149077 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:40.149570 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 02:31:40.149856 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 02:31:40.151335 13484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:40.253958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13329
I20260620 02:31:40.277478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13493
I20260620 02:31:40.292129 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:40.278393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.0.97.2:37421)
0620 02:31:40.278611 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:40.278618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:40.278639 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:31:40.278849 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:40.278861 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:40.278929 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:40.279041 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:40.280434 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.281211 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.282535 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.282955 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.283095 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:40.286032 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:40.286048 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:40.286064 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:40.286108 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:40.289124 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:40.289897 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:40.289914 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:40.289916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:40.290089 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:40.290508 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:40.290513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:40.290517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:40.290894 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:40.291214 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:40.291527 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 02:31:40.291757 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:31:40.292840 13628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:40.395222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13493
I20260620 02:31:40.413421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13637
I20260620 02:31:40.427385 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:40.414308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.0.97.3:43107)
0620 02:31:40.414531 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:40.414536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:40.414555 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:31:40.414614 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:40.414617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:40.414690 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:40.414799 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:40.416095 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.416909 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.418325 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:40.418773 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:40.418882 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:40.421620 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:40.421646 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:40.421663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:40.421699 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:40.424599 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:40.425419 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:40.425426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:40.425429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:40.425579 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:40.425956 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:40.425959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:40.425961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:40.426332 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:40.426558 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:40.426834 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 02:31:40.427005 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:31:40.428079 13762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:40.518337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13637
I20260620 02:31:40.535665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13259
I20260620 02:31:40.537032 13322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:40.649405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13259
I20260620 02:31:40.677155 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:31:41.013185 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:31:41.013401 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:31:41.359582 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:31:41.359 UTC [12311] LOG: received smart shutdown request
2026-06-20 02:31:41.363 UTC [12311] LOG: background worker "logical replication launcher" (PID 12320) exited with exit code 1
2026-06-20 02:31:41.363 UTC [12315] LOG: shutting down
2026-06-20 02:31:41.585 UTC [12311] LOG: database system is shut down
2026-06-20T02:31:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55756 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:31:41 dist-test-slave-r706 krb5kdc[13840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:31:41 dist-test-slave-r706 krb5kdc[13840](info): set up 2 sockets
Jun 20 02:31:41 dist-test-slave-r706 krb5kdc[13840](info): commencing operation
krb5kdc: starting...
W20260620 02:31:42.874567 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:31:43 dist-test-slave-r706 krb5kdc[13840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:31:43.153388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33495
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33495,127.0.97.61:43817,127.0.97.60:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:43.493141 13853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:43.493474 13853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:43.493568 13853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:43.501040 13853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:43.501168 13853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:43.501214 13853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:43.501273 13853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:43.501325 13853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:43.512046 13853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33495,127.0.97.61:43817,127.0.97.60:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:43.513437 13853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:43.514739 13853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:43.527320 13858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:43.527426 13859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:43.527320 13861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:43.528307 13853 server_base.cc:1034] running on GCE node
I20260620 02:31:43.529225 13853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:43.530795 13853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:43.532088 13853 hybrid_clock.cc:648] HybridClock initialized: now 1781922703532048 us; error 62 us; skew 500 ppm
I20260620 02:31:43.532533 13853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:43 dist-test-slave-r706 krb5kdc[13840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:43.541492 13853 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:43.543387 13853 webserver.cc:460] Webserver started at http://127.0.97.62:39475/ using document root <none> and password file <none>
I20260620 02:31:43.544193 13853 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:43.544375 13853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:43.544741 13853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:43.548184 13853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "2813f9cfb0e84c42a6650428dcf1739e"
format_stamp: "Formatted at 2026-06-20 02:31:43 on dist-test-slave-r706"
server_key: "1101f579b63fba723dcbde44bf37df08"
server_key_iv: "91b05394082b7556a4d8993dc0d35956"
server_key_version: "encryptionkey@0"
I20260620 02:31:43.549207 13853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "2813f9cfb0e84c42a6650428dcf1739e"
format_stamp: "Formatted at 2026-06-20 02:31:43 on dist-test-slave-r706"
server_key: "1101f579b63fba723dcbde44bf37df08"
server_key_iv: "91b05394082b7556a4d8993dc0d35956"
server_key_version: "encryptionkey@0"
I20260620 02:31:43.555260 13853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:31:43.559863 13868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:43.561874 13853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:31:43.562104 13853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "2813f9cfb0e84c42a6650428dcf1739e"
format_stamp: "Formatted at 2026-06-20 02:31:43 on dist-test-slave-r706"
server_key: "1101f579b63fba723dcbde44bf37df08"
server_key_iv: "91b05394082b7556a4d8993dc0d35956"
server_key_version: "encryptionkey@0"
I20260620 02:31:43.562356 13853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:43.597479 13853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:43.602159 13853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:43.602537 13853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:43.622887 13853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33495
I20260620 02:31:43.622926 13919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33495 every 8 connection(s)
I20260620 02:31:43.624321 13853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:43.628141 13920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:43.630028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13853
I20260620 02:31:43.630273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:31:43.631125 388 external_mini_cluster.cc:1428] Setting key 3b2bdf539c15905817e1f46e951df522
I20260620 02:31:43.638342 13920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:31:43 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 02:31:43.666761 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.642388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.97.62:33495)
0620 02:31:43.644542 (+ 2154us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:43.644548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:43.644592 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:31:43.646994 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:43.647001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:43.647241 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:43.647411 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:43.649165 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:43.650367 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:43.652751 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:43.653392 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:43.653562 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:43.658182 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:43.658192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:43.658207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:43.658245 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:43.661828 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:43.662655 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:43.662662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:43.662664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:43.662822 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:43.663243 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:43.663255 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:43.663258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:43.663808 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:43.664045 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:43.664717 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 02:31:43.665145 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1947,"spinlock_wait_cycles":62848,"thread_start_us":1677,"threads_started":1}
I20260620 02:31:43.666787 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.642388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33495 (local address 127.0.0.1:46250)
0620 02:31:43.643384 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.643436 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:43.646575 (+ 3139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:43.647489 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:43.647499 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:43.648401 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:43.648932 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:43.648961 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:43.650538 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:43.650546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:43.652605 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:43.652622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:43.653660 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:43.653672 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:43.653928 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:43.654729 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:43.654768 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:43.657890 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:43.662007 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:43.662021 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:43.662036 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:43.662486 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:43.662944 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:43.662951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:43.662955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:43.663121 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:43.663979 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:43.663987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:43.664282 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 02:31:43.664580 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 02:31:43.665274 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":204,"thread_start_us":428,"threads_started":1}
I20260620 02:31:43.671152 13920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:31:43.672510 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.672071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37344)
0620 02:31:43.672266 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.672385 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 02:31:43.672899 13872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
I20260620 02:31:43.674619 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.674243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37348)
0620 02:31:43.674385 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.674462 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:31:43.674860 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43817: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:31:43.692625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43817
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33495,127.0.97.61:43817,127.0.97.60:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:31:43.724254 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } attempt: 1
I20260620 02:31:43.725621 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.725081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37352)
0620 02:31:43.725319 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.725468 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 02:31:43.726821 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.726409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37354)
0620 02:31:43.726610 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.726683 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 02:31:43.727073 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43817: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
I20260620 02:31:43.795378 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } attempt: 2
I20260620 02:31:43.796676 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.796161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37356)
0620 02:31:43.796449 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.796535 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:31:43.797780 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.797423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37368)
0620 02:31:43.797572 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.797660 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:31:43.798023 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43817: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
I20260620 02:31:43.889300 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } attempt: 3
I20260620 02:31:43.890513 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.890063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37380)
0620 02:31:43.890269 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.890367 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:31:43.891777 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:43.891363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37384)
0620 02:31:43.891533 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:43.891624 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:31:43.892041 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43817: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
I20260620 02:31:44.035308 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } attempt: 4
I20260620 02:31:44.036592 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.036013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37386)
0620 02:31:44.036356 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.036459 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260620 02:31:44.037710 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.037363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37400)
0620 02:31:44.037512 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.037599 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 02:31:44.037982 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43817: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43817: connect: Connection refused (error 111)
W20260620 02:31:44.037930 13927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:44.038316 13927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:44.038481 13927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:44.046108 13927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:44.046239 13927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:44.046348 13927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:44.046452 13927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:44.046531 13927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:44.056795 13927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33495,127.0.97.61:43817,127.0.97.60:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:44.058288 13927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:44.059703 13927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:44.073140 13932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:44.072979 13935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:44.073096 13933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:44.074164 13927 server_base.cc:1034] running on GCE node
I20260620 02:31:44.075058 13927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:44.076754 13927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:44.077970 13927 hybrid_clock.cc:648] HybridClock initialized: now 1781922704077896 us; error 75 us; skew 500 ppm
I20260620 02:31:44.078367 13927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:44.081835 13927 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:31:44.083541 13927 webserver.cc:460] Webserver started at http://127.0.97.61:38825/ using document root <none> and password file <none>
I20260620 02:31:44.084306 13927 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:44.084487 13927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:44.084817 13927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:44.088084 13927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data/instance:
uuid: "acc73b40d99143bca9a3aef65da7dcba"
format_stamp: "Formatted at 2026-06-20 02:31:44 on dist-test-slave-r706"
server_key: "b5886f571b897337b6a354234d4bd16c"
server_key_iv: "aa3214954488aa55400b82f0d6929704"
server_key_version: "encryptionkey@0"
I20260620 02:31:44.089044 13927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal/instance:
uuid: "acc73b40d99143bca9a3aef65da7dcba"
format_stamp: "Formatted at 2026-06-20 02:31:44 on dist-test-slave-r706"
server_key: "b5886f571b897337b6a354234d4bd16c"
server_key_iv: "aa3214954488aa55400b82f0d6929704"
server_key_version: "encryptionkey@0"
I20260620 02:31:44.095247 13927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:31:44.099296 13942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:44.100796 13927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:31:44.101083 13927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal
uuid: "acc73b40d99143bca9a3aef65da7dcba"
format_stamp: "Formatted at 2026-06-20 02:31:44 on dist-test-slave-r706"
server_key: "b5886f571b897337b6a354234d4bd16c"
server_key_iv: "aa3214954488aa55400b82f0d6929704"
server_key_version: "encryptionkey@0"
I20260620 02:31:44.101329 13927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:44.117928 13927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:44.122948 13927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:44.123329 13927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:44.144644 13927 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43817
I20260620 02:31:44.144666 13993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43817 every 8 connection(s)
I20260620 02:31:44.146271 13927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:31:44.148324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 13927
I20260620 02:31:44.148526 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-1/wal/instance
I20260620 02:31:44.149106 388 external_mini_cluster.cc:1428] Setting key 9fa2457d31a3591d9c897e096761fb46
I20260620 02:31:44.150980 13994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:44.160827 13994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:44.187582 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.163860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.0.97.62:33495)
0620 02:31:44.164131 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.164142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.165539 (+ 1397us) server_negotiation.cc:407] Connection header received
0620 02:31:44.168955 (+ 3416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.168960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.169111 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.169202 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.171632 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.172321 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.175089 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.175577 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.175690 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.179690 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.179702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.179711 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.179768 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.183950 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.184928 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.184937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.184944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.185157 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.185657 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.185665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.185671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.186142 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.186305 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.187118 (+ 813us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.187316 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:31:44.189087 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.164271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33495 (local address 127.0.0.1:46260)
0620 02:31:44.165325 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.165385 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.168546 (+ 3161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.169556 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.169573 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.170452 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.171411 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.171446 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.172513 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.172521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.174848 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.174881 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.175835 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.175841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.176074 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.176841 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.176868 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.179381 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.184164 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.184173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.184189 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.184746 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.185320 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.185324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.185327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.185523 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.186303 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.186310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.186683 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.186926 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.187579 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":393,"threads_started":1}
I20260620 02:31:44.190950 13994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:31:44.206499 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.191735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37408)
0620 02:31:44.191942 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.191975 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.192094 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.193192 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.193196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.193655 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.193784 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.193792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.195129 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.195135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.195981 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.195992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.196838 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.196844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.197101 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.197813 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.197831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.200200 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.203727 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.203735 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.203739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.204206 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.204594 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.204601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.204606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.204780 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.205535 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.205545 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.205699 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.205872 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.206188 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 02:31:44.206859 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.191968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.0.97.61:43817)
0620 02:31:44.192712 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.192718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.192743 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:31:44.192787 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.192794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.192973 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.193213 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.193938 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.194976 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.196114 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.196665 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.196798 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.200437 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.200451 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.200467 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.200501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.203576 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.204349 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.204362 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.204364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.204473 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.204878 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.204884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.204889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.205391 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.205613 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.206182 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.206410 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":453,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:31:44.209451 13994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:31:44.210829 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.210319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54840)
0620 02:31:44.210571 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.210690 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 02:31:44.210829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33437
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33495,127.0.97.61:43817,127.0.97.60:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:44.211212 13944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.212844 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.212526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54846)
0620 02:31:44.212670 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.212751 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:31:44.213086 13994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.262449 13994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 1
I20260620 02:31:44.264008 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.263329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54862)
0620 02:31:44.263618 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.263771 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:31:44.265357 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.264856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54868)
0620 02:31:44.265091 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.265203 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 02:31:44.265643 13994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.333940 13994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 2
I20260620 02:31:44.335182 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.334712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54882)
0620 02:31:44.334910 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.335015 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:31:44.336210 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.335865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54888)
0620 02:31:44.336017 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.336096 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:31:44.336414 13994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.337218 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } attempt: 5
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:31:44.351023 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.337893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.0.97.61:43817)
0620 02:31:44.338104 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.338112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.338140 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:31:44.338212 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.338220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.338334 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.338474 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.339228 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.340308 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.341630 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.342106 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.342241 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.345719 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.345724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.345727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.345760 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.348297 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.348964 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.348971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.348976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.349173 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.349590 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.349597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.349600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.349892 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.350070 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.350457 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.350688 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:31:44.351033 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.337811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37422)
0620 02:31:44.337987 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.338011 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.338110 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.338489 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.338492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.338871 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.339068 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.339077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.340448 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.340452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.341453 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.341469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.342272 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.342279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.342478 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.343162 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.343180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.345521 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.348453 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.348458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.348463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.348842 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.349287 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.349293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.349297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.349439 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.350040 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.350045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.350210 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.350356 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.350666 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 02:31:44.352226 13920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:31:44.353363 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.352911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54900)
0620 02:31:44.353129 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.353224 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 02:31:44.354447 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.354152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54914)
0620 02:31:44.354288 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.354354 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 02:31:44.354743 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.405972 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 1
I20260620 02:31:44.407395 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.406927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54926)
0620 02:31:44.407134 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.407224 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 02:31:44.408377 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.408043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54930)
0620 02:31:44.408207 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.408280 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 02:31:44.408603 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.427649 13994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 3
I20260620 02:31:44.428861 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.428361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54934)
0620 02:31:44.428609 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.428705 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:31:44.429991 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.429562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54946)
0620 02:31:44.429795 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.429863 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260620 02:31:44.430193 13994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.476855 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 2
I20260620 02:31:44.478272 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.477726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54960)
0620 02:31:44.478002 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.478096 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 02:31:44.479336 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.478986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54968)
0620 02:31:44.479143 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.479226 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:31:44.479561 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
W20260620 02:31:44.554205 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:44.554541 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:44.554719 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:44.562160 14001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:44.562319 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:44.562434 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:44.562534 14001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:44.562619 14001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:44.573316 14001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33495,127.0.97.61:43817,127.0.97.60:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:44.573426 13994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 4
I20260620 02:31:44.574783 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.574102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54982)
0620 02:31:44.574535 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.574628 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260620 02:31:44.574867 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:44.575795 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.575465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:54996)
0620 02:31:44.575610 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.575692 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 02:31:44.576004 13994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
I20260620 02:31:44.576332 14001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:31:44.585798 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 3
I20260620 02:31:44.587121 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.586528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:55006)
0620 02:31:44.586765 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.586891 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:31:44.588606 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.587976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:55014)
0620 02:31:44.588314 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.588420 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
W20260620 02:31:44.588905 13920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33437: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33437: connect: Connection refused (error 111)
W20260620 02:31:44.590302 14006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:44.590359 14007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:44.590338 14009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:44.591224 14001 server_base.cc:1034] running on GCE node
I20260620 02:31:44.592247 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:44.593922 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:44.595204 14001 hybrid_clock.cc:648] HybridClock initialized: now 1781922704595146 us; error 47 us; skew 500 ppm
I20260620 02:31:44.595619 14001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:44.599181 14001 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:31:44.600841 14001 webserver.cc:460] Webserver started at http://127.0.97.60:43735/ using document root <none> and password file <none>
I20260620 02:31:44.601610 14001 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:44.601744 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:44.602084 14001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:44.605448 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data/instance:
uuid: "43cbd34467a14c6893f28c6806e4e84d"
format_stamp: "Formatted at 2026-06-20 02:31:44 on dist-test-slave-r706"
server_key: "12b716e4e2e0da501a0300d8d6438d2a"
server_key_iv: "384f7a2501849316159e7c6ef21b9f98"
server_key_version: "encryptionkey@0"
I20260620 02:31:44.606348 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal/instance:
uuid: "43cbd34467a14c6893f28c6806e4e84d"
format_stamp: "Formatted at 2026-06-20 02:31:44 on dist-test-slave-r706"
server_key: "12b716e4e2e0da501a0300d8d6438d2a"
server_key_iv: "384f7a2501849316159e7c6ef21b9f98"
server_key_version: "encryptionkey@0"
I20260620 02:31:44.611898 14001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:31:44.615478 14016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:44.616885 14001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:31:44.617097 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal
uuid: "43cbd34467a14c6893f28c6806e4e84d"
format_stamp: "Formatted at 2026-06-20 02:31:44 on dist-test-slave-r706"
server_key: "12b716e4e2e0da501a0300d8d6438d2a"
server_key_iv: "384f7a2501849316159e7c6ef21b9f98"
server_key_version: "encryptionkey@0"
I20260620 02:31:44.617309 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:44.636197 14001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:44.640640 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:44.640962 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:44.661870 14001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33437
I20260620 02:31:44.661871 14067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33437 every 8 connection(s)
I20260620 02:31:44.663582 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:31:44.666584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14001
I20260620 02:31:44.666844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922485472743-388-0/minicluster-data/master-2/wal/instance
I20260620 02:31:44.667529 388 external_mini_cluster.cc:1428] Setting key 389d3ccec8caf07a30292af2fc69a700
I20260620 02:31:44.669740 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:44.677987 14068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:44.691905 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.669904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.0.97.60:33437)
0620 02:31:44.670778 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.670790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.670833 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:31:44.672433 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.672455 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.672889 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.673410 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.674975 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.676456 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.678015 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.678753 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.678959 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.684022 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.684030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.684041 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.684078 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.687228 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.688072 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.688084 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.688097 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.688327 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.688768 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.688777 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.688780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.689507 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.689820 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.690281 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.690726 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":383,"threads_started":1}
I20260620 02:31:44.696792 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.680897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.97.62:33495)
0620 02:31:44.681133 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.681137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.682020 (+ 883us) server_negotiation.cc:407] Connection header received
0620 02:31:44.682280 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.682286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.682359 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.682481 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.683794 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.684626 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.686320 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.686783 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.686961 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.690813 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.690819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.690823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.690858 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.693717 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.694606 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.694612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.694616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.694768 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.695245 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.695250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.695253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.695667 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.695843 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.696281 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.696540 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:31:44.697129 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.680909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33495 (local address 127.0.0.1:46270)
0620 02:31:44.681849 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.681895 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.682115 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.682568 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.682576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.683298 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.683612 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.683622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.684814 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.684819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.686121 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.686140 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.687013 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.687021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.687234 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.687900 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.687924 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.690587 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.693850 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.693859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.693865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.694451 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.694917 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.694922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.694925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.695109 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.695800 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.695806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.695971 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.696118 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.696478 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":546,"threads_started":1}
I20260620 02:31:44.698875 14068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:31:44.713043 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.695314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.0.97.62:33495)
0620 02:31:44.695951 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.695954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.695972 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:31:44.696007 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.696010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.696060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.696183 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.697951 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.699111 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.700786 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.701336 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.701468 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.706863 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.706868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.706872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.706907 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.709776 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.710739 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.710744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.710746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.710884 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.711344 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.711349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.711351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.711672 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.711898 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.712257 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.712562 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":270,"threads_started":1}
I20260620 02:31:44.714597 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.699806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43817 (local address 127.0.0.1:37430)
0620 02:31:44.699993 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.700012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.700109 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.700460 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.700476 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.700905 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.701106 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.701115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.702379 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.702385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.703892 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.703911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.704738 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.704745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.704914 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.705670 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.705698 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.708139 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.711695 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.711700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.711704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.712258 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.712755 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.712760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.712765 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.712948 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.713630 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.713636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.713790 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.713932 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.714232 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 02:31:44.714648 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.699881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.0.97.61:43817)
0620 02:31:44.700062 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.700066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.700084 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:31:44.700255 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.700259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.700324 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.700407 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.701284 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.702217 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.704073 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.704555 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.704671 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.708360 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.708366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.708370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.708415 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.711558 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.712395 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.712401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.712404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.712593 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.713138 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.713145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.713148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.713499 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.713671 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.714048 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.714271 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:31:44.716051 14068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:31:44.729414 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.716740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:55040)
0620 02:31:44.716900 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.716919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.717049 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.717426 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.717430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.717824 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.717947 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.717955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.718969 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.718977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.719936 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.719947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.720695 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.720700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.720896 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.721594 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.721611 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.723882 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.726878 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.726885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.726889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.727251 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.727625 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.727632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.727635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.727762 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.728299 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.728303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.728442 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.728580 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.728922 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":8}
I20260620 02:31:44.729470 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.717041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.0.97.60:33437)
0620 02:31:44.717212 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.717216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.717228 (+ 12us) server_negotiation.cc:407] Connection header received
0620 02:31:44.717261 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.717264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.717310 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.717410 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.718081 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.718849 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.720081 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.720549 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.720684 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.724048 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.724053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.724078 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.724119 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.726727 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.727365 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.727372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.727375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.727519 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.727863 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.727870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.727875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.728170 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.728325 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.728659 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.729121 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":59}
I20260620 02:31:44.735954 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.714730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.0.97.61:43817)
0620 02:31:44.714927 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.714933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.715039 (+ 106us) server_negotiation.cc:407] Connection header received
0620 02:31:44.715305 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.715313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.715415 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.715535 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.717896 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.719443 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.722065 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.722995 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.723115 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.729147 (+ 6032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.729152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.729155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.729189 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.732749 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.733625 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.733632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.733636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.733813 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.734215 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.734219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.734222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.734601 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.734891 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.735166 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.735590 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":12317,"threads_started":1}
I20260620 02:31:44.735992 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: Bootstrap starting.
I20260620 02:31:44.740159 14068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:44.741469 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:31:44.741585 14068 log.cc:824] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:44.744390 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: No bootstrap required, opened a new log
I20260620 02:31:44.755427 14068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.755937 14068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:44.756058 14068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cbd34467a14c6893f28c6806e4e84d, State: Initialized, Role: FOLLOWER
I20260620 02:31:44.756932 14068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.764271 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.764803 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:44.766158 13920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 4
I20260620 02:31:44.766983 14068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:44.770195 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.747171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.0.97.60:33437)
0620 02:31:44.748026 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.748034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.748054 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:31:44.748288 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.748294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.748366 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.748483 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.753266 (+ 4783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.754324 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.756586 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.757398 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.757497 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.762007 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.762015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.762020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.762079 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.766295 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.767399 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.767405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.767411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.767598 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.768082 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.768087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.768091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.768504 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.768738 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.769514 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.769796 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704}
I20260620 02:31:44.770375 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.746682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.0.97.62:33495)
0620 02:31:44.746894 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.746900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.750388 (+ 3488us) server_negotiation.cc:407] Connection header received
0620 02:31:44.750541 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.750547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.750647 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.750761 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.752577 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.753682 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.756060 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.756744 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.756835 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.762007 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.762015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.762020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.762074 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.766845 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.767780 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.767785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.767788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.767935 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.768407 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.768412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.768415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.768791 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.769044 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.769789 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.770033 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:31:44.771979 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:31:44.772312 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.774894 14083 leader_election.cc:288] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2813f9cfb0e84c42a6650428dcf1739e (127.0.97.62:33495), acc73b40d99143bca9a3aef65da7dcba (127.0.97.61:43817)
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:31:44.785729 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.746683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.0.97.61:43817)
0620 02:31:44.746891 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.746897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.749661 (+ 2764us) server_negotiation.cc:407] Connection header received
0620 02:31:44.749834 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.749841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.749941 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.750058 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.752191 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.753440 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.755756 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.756451 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.756559 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.762450 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.762461 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.762467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.762544 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.768565 (+ 6021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.779947 (+ 11382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.779955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.779961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.780494 (+ 533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.781999 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.782005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.782008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.782447 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.782797 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.784627 (+ 1830us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.785325 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:31:44.791365 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.767554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.0.97.60:33437)
0620 02:31:44.770530 (+ 2976us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.770537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.770556 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:31:44.770607 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.770611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.770707 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.771595 (+ 888us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.771757 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.772785 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.776041 (+ 3256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.776754 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.778342 (+ 1588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.782590 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.782599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.782615 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.782660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.787988 (+ 5328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.788973 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.788979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.788983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.789201 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.789628 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.789635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.789638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.790087 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.790241 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.790700 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.790939 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2852,"thread_start_us":15029,"threads_started":1}
I20260620 02:31:44.791360 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.766802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:55058)
0620 02:31:44.767211 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.767232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.767339 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.770886 (+ 3547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.770892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.771299 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.771467 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.771476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.772951 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.772956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.774184 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.774200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.776953 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.776961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.777579 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.778946 (+ 1367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.778976 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.782297 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.788158 (+ 5861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.788164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.788169 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.788799 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.789324 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.789329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.789333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.789489 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.790267 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.790273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.790430 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.790605 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.790945 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260620 02:31:44.799945 13920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: Bootstrap starting.
W20260620 02:31:44.804030 14095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:31:44.804181 14095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:31:44.806444 13920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:44.807912 13920 log.cc:824] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:44.799342 14001 master_runner.cc:186] Error getting master registration for 127.0.97.62:33495: OK, instance_id { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" instance_seqno: 1781922703617148 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:31:44.811345 14001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:31:44.813967 13920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: No bootstrap required, opened a new log
I20260620 02:31:44.827250 13920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.827719 13920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:44.827944 13920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2813f9cfb0e84c42a6650428dcf1739e, State: Initialized, Role: FOLLOWER
I20260620 02:31:44.828786 13920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.830205 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.830667 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:44.831652 13920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:44.831707 13895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43cbd34467a14c6893f28c6806e4e84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f9cfb0e84c42a6650428dcf1739e" is_pre_election: true
I20260620 02:31:44.832278 13895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43cbd34467a14c6893f28c6806e4e84d in term 0.
I20260620 02:31:44.832949 14019 leader_election.cc:302] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f9cfb0e84c42a6650428dcf1739e, 43cbd34467a14c6893f28c6806e4e84d; no voters:
I20260620 02:31:44.833448 14083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:31:44.833626 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:31:44.833758 14083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:44.836122 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.837399 14083 leader_election.cc:288] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [CANDIDATE]: Term 1 election: Requested vote from peers 2813f9cfb0e84c42a6650428dcf1739e (127.0.97.62:33495), acc73b40d99143bca9a3aef65da7dcba (127.0.97.61:43817)
I20260620 02:31:44.838320 13895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43cbd34467a14c6893f28c6806e4e84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2813f9cfb0e84c42a6650428dcf1739e"
I20260620 02:31:44.838564 13895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:44.842360 13895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43cbd34467a14c6893f28c6806e4e84d in term 1.
I20260620 02:31:44.842901 14019 leader_election.cc:302] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2813f9cfb0e84c42a6650428dcf1739e, 43cbd34467a14c6893f28c6806e4e84d; no voters:
I20260620 02:31:44.843293 14083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:44.843896 14083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [term 1 LEADER]: Becoming Leader. State: Replica: 43cbd34467a14c6893f28c6806e4e84d, State: Running, Role: LEADER
I20260620 02:31:44.844491 14083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
W20260620 02:31:44.855734 13874 tablet.cc:2369] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:44.859717 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43cbd34467a14c6893f28c6806e4e84d. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.860219 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:44.861524 14108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:44.863970 14108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:44.875182 13994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } attempt: 5
I20260620 02:31:44.876921 14113 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:31:44.879626 13895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [term 1 FOLLOWER]: Refusing update from remote peer 43cbd34467a14c6893f28c6806e4e84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:44.880903 14104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 20 02:31:44 dist-test-slave-r706 krb5kdc[13840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 02:31:44.893316 14022 tablet.cc:2369] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:44.884841 13853 master_runner.cc:186] Error getting master registration for 127.0.97.61:43817: OK, instance_id { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" instance_seqno: 1781922704138651 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:31:44.896124 13853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:31:44.903708 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.878328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.0.97.60:33437)
0620 02:31:44.878533 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:44.878539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:44.878560 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:31:44.878619 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:44.878623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:44.878710 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:44.878842 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:44.880170 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.881280 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.883218 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.883909 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.884042 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.888649 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:44.888657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:44.888662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:44.888715 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:44.899946 (+ 11231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.901129 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.901136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.901140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.901327 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:44.901832 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:44.901836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:44.901841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:44.902248 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:44.902387 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:44.902962 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 02:31:44.903294 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:31:44.904778 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:44.875784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33437 (local address 127.0.0.1:55062)
0620 02:31:44.876135 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:44.876156 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:44.876277 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:44.879176 (+ 2899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:44.879181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:44.879720 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:44.879956 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.879970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.881502 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.881509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.882980 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:44.883007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:44.884332 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:44.884339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:44.884550 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:44.885392 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:44.885417 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:44.888412 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:44.900234 (+ 11822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.900244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.900249 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.900922 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.901501 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:44.901506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:44.901510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:44.901692 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:44.902488 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:44.902493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:44.902671 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 02:31:44.904021 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0620 02:31:44.904373 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260620 02:31:44.911957 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: Bootstrap starting.
I20260620 02:31:44.914112 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43cbd34467a14c6893f28c6806e4e84d. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.914486 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:44.914438 14112 mvcc.cc:205] Tried to move back new op lower bound from 7298755399158964224 to 7298755399047925760. Current Snapshot: MvccSnapshot[applied={T|T < 7298755399158964224}]
I20260620 02:31:44.916157 13994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:44.918242 13994 log.cc:824] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:44.920873 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.921277 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:44.927855 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.928166 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: No bootstrap required, opened a new log
I20260620 02:31:44.928184 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:44.928323 14108 catalog_manager.cc:1347] Generated new cluster ID: ed0d0402c2554a3dbe3e3e31694e6bf6
I20260620 02:31:44.928427 14108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:31:44.941587 13994 raft_consensus.cc:348] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.942080 13994 raft_consensus.cc:374] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:44.942266 13994 raft_consensus.cc:729] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc73b40d99143bca9a3aef65da7dcba, State: Initialized, Role: FOLLOWER
I20260620 02:31:44.943002 13994 consensus_queue.cc:260] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } }
I20260620 02:31:44.944098 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:44.944621 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:44.945531 13994 sys_catalog.cc:564] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:44.945591 13969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43cbd34467a14c6893f28c6806e4e84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc73b40d99143bca9a3aef65da7dcba" is_pre_election: true
I20260620 02:31:44.945930 13967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:44.947233 13968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43cbd34467a14c6893f28c6806e4e84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc73b40d99143bca9a3aef65da7dcba"
I20260620 02:31:44.951023 13967 raft_consensus.cc:1261] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [term 1 FOLLOWER]: Refusing update from remote peer 43cbd34467a14c6893f28c6806e4e84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:44.952252 14104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:31:44.977746 14108 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:31:44.980370 14108 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:31:45.005865 14108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d: Generated new TSK 0
I20260620 02:31:45.007376 14108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:31:45.015002 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43cbd34467a14c6893f28c6806e4e84d. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:45.015378 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:45.020133 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:45.020496 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:45.021224 14129 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43cbd34467a14c6893f28c6806e4e84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2813f9cfb0e84c42a6650428dcf1739e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33495 } } peers { permanent_uuid: "acc73b40d99143bca9a3aef65da7dcba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43817 } } peers { permanent_uuid: "43cbd34467a14c6893f28c6806e4e84d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33437 } } }
I20260620 02:31:45.028244 14129 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:31:45.062408 14138 catalog_manager.cc:1259] Loaded cluster ID: ed0d0402c2554a3dbe3e3e31694e6bf6
I20260620 02:31:45.062553 14138 catalog_manager.cc:1552] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: loading cluster ID for follower catalog manager: success
I20260620 02:31:45.067641 14138 catalog_manager.cc:1574] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: acquiring CA information for follower catalog manager: success
I20260620 02:31:45.070734 14138 catalog_manager.cc:1602] T 00000000000000000000000000000000 P acc73b40d99143bca9a3aef65da7dcba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:31:45.882315 14113 catalog_manager.cc:1259] Loaded cluster ID: ed0d0402c2554a3dbe3e3e31694e6bf6
I20260620 02:31:45.882457 14113 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: loading cluster ID for follower catalog manager: success
I20260620 02:31:45.885727 14113 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: acquiring CA information for follower catalog manager: success
I20260620 02:31:45.887818 14113 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2813f9cfb0e84c42a6650428dcf1739e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:31:46.137722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13853
I20260620 02:31:46.139016 13915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:46.259995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13853
I20260620 02:31:46.279373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 13927
I20260620 02:31:46.280471 13989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:46.347638 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:46.346704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33495 (local address 127.0.0.1:46288)
0620 02:31:46.347172 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:46.347327 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":252,"threads_started":1}
W20260620 02:31:46.348207 14019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33495: connect: Connection refused (error 111)
I20260620 02:31:46.350178 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:46.349773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33495 (local address 127.0.0.1:46292)
0620 02:31:46.349942 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:46.350035 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:31:46.350430 14019 consensus_peers.cc:487] T 00000000000000000000000000000000 P 43cbd34467a14c6893f28c6806e4e84d -> Peer 2813f9cfb0e84c42a6650428dcf1739e (127.0.97.62:33495): Couldn't send request to peer 2813f9cfb0e84c42a6650428dcf1739e. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:31:46.392198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 13927
I20260620 02:31:46.414577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14001
I20260620 02:31:46.415755 14063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:46.527194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14001
2026-06-20T02:31:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4791 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:31:46 dist-test-slave-r706 krb5kdc[14150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:31:46 dist-test-slave-r706 krb5kdc[14150](info): set up 2 sockets
Jun 20 02:31:46 dist-test-slave-r706 krb5kdc[14150](info): commencing operation
krb5kdc: starting...
W20260620 02:31:47.653754 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:31:47 dist-test-slave-r706 krb5kdc[14150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:31:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:31:47.921177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45081
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:48.269642 14163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:48.270064 14163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:48.270248 14163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:48.278117 14163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:48.278347 14163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:48.278475 14163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:48.278587 14163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:48.278680 14163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:48.289881 14163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:48.291380 14163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:48.292775 14163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:48.305497 14168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:48.305459 14171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:48.305541 14169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:48.306337 14163 server_base.cc:1034] running on GCE node
I20260620 02:31:48.307188 14163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:48.308717 14163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:48.309965 14163 hybrid_clock.cc:648] HybridClock initialized: now 1781922708309898 us; error 91 us; skew 500 ppm
I20260620 02:31:48.310340 14163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:48 dist-test-slave-r706 krb5kdc[14150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:48.313820 14163 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:48.315681 14163 webserver.cc:460] Webserver started at http://127.0.97.62:33959/ using document root <none> and password file <none>
I20260620 02:31:48.316422 14163 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:48.316573 14163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:48.316901 14163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:48.320091 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "f88166de681245a0a4802ef0ab58602a"
format_stamp: "Formatted at 2026-06-20 02:31:48 on dist-test-slave-r706"
server_key: "d76741ae7f2e8f7522eaa55b34749af4"
server_key_iv: "770fab9f72c0354f87abc3d57ff398dd"
server_key_version: "encryptionkey@0"
I20260620 02:31:48.320904 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "f88166de681245a0a4802ef0ab58602a"
format_stamp: "Formatted at 2026-06-20 02:31:48 on dist-test-slave-r706"
server_key: "d76741ae7f2e8f7522eaa55b34749af4"
server_key_iv: "770fab9f72c0354f87abc3d57ff398dd"
server_key_version: "encryptionkey@0"
I20260620 02:31:48.326408 14163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:31:48.330133 14178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:48.331494 14163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:31:48.331693 14163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "f88166de681245a0a4802ef0ab58602a"
format_stamp: "Formatted at 2026-06-20 02:31:48 on dist-test-slave-r706"
server_key: "d76741ae7f2e8f7522eaa55b34749af4"
server_key_iv: "770fab9f72c0354f87abc3d57ff398dd"
server_key_version: "encryptionkey@0"
I20260620 02:31:48.331911 14163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:48.344846 14163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:48.349526 14163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:48.349833 14163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:48.370009 14163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45081
I20260620 02:31:48.370033 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45081 every 8 connection(s)
I20260620 02:31:48.371520 14163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:48.375561 14230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:48.377596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14163
I20260620 02:31:48.377842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:31:48.378711 388 external_mini_cluster.cc:1428] Setting key fd4d6b845504a55f08c08f711e5eb0de
I20260620 02:31:48.389305 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a: Bootstrap starting.
I20260620 02:31:48.393463 14230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:48.394862 14230 log.cc:824] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:31:48 dist-test-slave-r706 krb5kdc[14150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:48.398144 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a: No bootstrap required, opened a new log
I20260620 02:31:48.404078 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:48.381160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.0.97.62:45081)
0620 02:31:48.382179 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:48.382193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:48.382246 (+ 53us) server_negotiation.cc:407] Connection header received
0620 02:31:48.383998 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:48.384024 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:48.384449 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:48.384784 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:48.386748 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:48.388119 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:48.390131 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:48.390868 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:48.391020 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:48.396349 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:48.396359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:48.396375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:48.396421 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:48.399652 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:48.400472 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:48.400479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:48.400488 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:48.400667 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:48.401133 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:48.401140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:48.401149 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:48.401718 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:48.402044 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:48.402565 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 02:31:48.403021 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":480,"threads_started":1}
I20260620 02:31:48.406455 14230 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88166de681245a0a4802ef0ab58602a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45081 } }
I20260620 02:31:48.406906 14230 raft_consensus.cc:374] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:48.407023 14230 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88166de681245a0a4802ef0ab58602a, State: Initialized, Role: FOLLOWER
I20260620 02:31:48.407694 14230 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88166de681245a0a4802ef0ab58602a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45081 } }
I20260620 02:31:48.408007 14230 raft_consensus.cc:388] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:31:48.408174 14230 raft_consensus.cc:482] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:31:48.408278 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:31:48.408368 14230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:48.411298 14230 raft_consensus.cc:504] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88166de681245a0a4802ef0ab58602a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45081 } }
I20260620 02:31:48.411945 14230 leader_election.cc:302] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f88166de681245a0a4802ef0ab58602a; no voters:
I20260620 02:31:48.412662 14230 leader_election.cc:288] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:31:48.412803 14235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:48.413921 14235 raft_consensus.cc:686] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [term 1 LEADER]: Becoming Leader. State: Replica: f88166de681245a0a4802ef0ab58602a, State: Running, Role: LEADER
I20260620 02:31:48.414588 14235 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88166de681245a0a4802ef0ab58602a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45081 } }
I20260620 02:31:48.415728 14230 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:48.417639 14236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88166de681245a0a4802ef0ab58602a. Latest consensus state: current_term: 1 leader_uuid: "f88166de681245a0a4802ef0ab58602a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88166de681245a0a4802ef0ab58602a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45081 } } }
I20260620 02:31:48.417667 14237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f88166de681245a0a4802ef0ab58602a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88166de681245a0a4802ef0ab58602a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45081 } } }
I20260620 02:31:48.418102 14236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:48.418251 14237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:48.419716 14241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:48.425082 14241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:48.433928 14241 catalog_manager.cc:1347] Generated new cluster ID: 255f25e1fe894f5a988190823c501247
I20260620 02:31:48.434055 14241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:31:48.471532 14241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:31:48.473106 14241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:31:48.482112 14241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f88166de681245a0a4802ef0ab58602a: Generated new TSK 0
I20260620 02:31:48.483124 14241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:31:48.522325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14163
I20260620 02:31:48.523650 14225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:48.621250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14163
2026-06-20T02:31:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2089 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:31:48 dist-test-slave-r706 krb5kdc[14258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:31:48 dist-test-slave-r706 krb5kdc[14258](info): set up 2 sockets
Jun 20 02:31:48 dist-test-slave-r706 krb5kdc[14258](info): commencing operation
krb5kdc: starting...
W20260620 02:31:49.753595 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:31:49 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:31:50.030015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44705
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:50.383534 14271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:50.383904 14271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:50.384065 14271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:50.391847 14271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:50.392006 14271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:50.392091 14271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:50.392198 14271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:50.392285 14271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:50.402503 14271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:50.403954 14271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:50.405336 14271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:50.417668 14276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:50.417693 14279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:50.417897 14277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:50.418596 14271 server_base.cc:1034] running on GCE node
I20260620 02:31:50.419358 14271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:50.420866 14271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:50.422130 14271 hybrid_clock.cc:648] HybridClock initialized: now 1781922710422093 us; error 81 us; skew 500 ppm
I20260620 02:31:50.422516 14271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:50 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:50.425866 14271 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:50.427573 14271 webserver.cc:460] Webserver started at http://127.0.97.62:35089/ using document root <none> and password file <none>
I20260620 02:31:50.428294 14271 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:50.428406 14271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:50.428670 14271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:50.431823 14271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "1485a16502cb49949e9518de3112c0a2"
format_stamp: "Formatted at 2026-06-20 02:31:50 on dist-test-slave-r706"
server_key: "7141c377b3b2ca43de3410aace0a1999"
server_key_iv: "86519d818ad34657fe9e7aaaeb5009e2"
server_key_version: "encryptionkey@0"
I20260620 02:31:50.432560 14271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "1485a16502cb49949e9518de3112c0a2"
format_stamp: "Formatted at 2026-06-20 02:31:50 on dist-test-slave-r706"
server_key: "7141c377b3b2ca43de3410aace0a1999"
server_key_iv: "86519d818ad34657fe9e7aaaeb5009e2"
server_key_version: "encryptionkey@0"
I20260620 02:31:50.437985 14271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 02:31:50.441725 14286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:50.443270 14271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:31:50.443450 14271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "1485a16502cb49949e9518de3112c0a2"
format_stamp: "Formatted at 2026-06-20 02:31:50 on dist-test-slave-r706"
server_key: "7141c377b3b2ca43de3410aace0a1999"
server_key_iv: "86519d818ad34657fe9e7aaaeb5009e2"
server_key_version: "encryptionkey@0"
I20260620 02:31:50.443645 14271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:50.499260 14271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:50.503741 14271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:50.504020 14271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:50.523491 14271 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44705
I20260620 02:31:50.523538 14337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44705 every 8 connection(s)
I20260620 02:31:50.524904 14271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:50.527441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14271
I20260620 02:31:50.527673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:31:50.528353 388 external_mini_cluster.cc:1428] Setting key 5b6be95d9998e069f41e3a80e42033b3
I20260620 02:31:50.529088 14338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:31:50 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:50.551645 14338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap starting.
I20260620 02:31:50.555341 14338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:50.556653 14338 log.cc:824] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:50.559736 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:50.530415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.97.62:44705)
0620 02:31:50.531226 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:50.531237 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:50.531281 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:31:50.533827 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:50.533853 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:50.534272 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:50.534646 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:50.536597 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:50.538073 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:50.539669 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:50.542247 (+ 2578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:50.543426 (+ 1179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:50.549069 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:50.549087 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:50.549110 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:50.549179 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:50.554357 (+ 5178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:50.555397 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:50.555410 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:50.555421 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:50.555663 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:50.556163 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:50.556171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:50.556177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:50.557080 (+ 903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:50.557469 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:50.558029 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 02:31:50.558392 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260620 02:31:50.560738 14338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: No bootstrap required, opened a new log
I20260620 02:31:50.569897 14338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:50.570322 14338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:50.570441 14338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1485a16502cb49949e9518de3112c0a2, State: Initialized, Role: FOLLOWER
I20260620 02:31:50.571108 14338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:50.571384 14338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:31:50.571538 14338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:31:50.571722 14338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:50.574203 14338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:50.574755 14338 leader_election.cc:302] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1485a16502cb49949e9518de3112c0a2; no voters:
I20260620 02:31:50.575539 14338 leader_election.cc:288] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:31:50.575670 14343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:50.577242 14343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1485a16502cb49949e9518de3112c0a2, State: Running, Role: LEADER
I20260620 02:31:50.577939 14343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:50.578404 14338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:50.580595 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1485a16502cb49949e9518de3112c0a2. Latest consensus state: current_term: 1 leader_uuid: "1485a16502cb49949e9518de3112c0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } } }
I20260620 02:31:50.581066 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:50.581846 14344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1485a16502cb49949e9518de3112c0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } } }
I20260620 02:31:50.582185 14344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:50.591158 14348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:50.599073 14348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:50.615134 14348 catalog_manager.cc:1347] Generated new cluster ID: 84c7ba71dd894e85a4e00e7d6b57ff49
I20260620 02:31:50.615320 14348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:31:50.643108 14348 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:31:50.644951 14348 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:31:50.652124 14348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Generated new TSK 0
I20260620 02:31:50.653131 14348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:31:50.717312 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44705
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:51.130116 14365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:51.130458 14365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:51.130568 14365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:51.138474 14365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:51.138615 14365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:51.138741 14365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:31:51.149288 14365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:51.150728 14365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:51.152469 14365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:51.167121 14370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:51.167095 14373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:51.167155 14371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:51.168484 14365 server_base.cc:1034] running on GCE node
I20260620 02:31:51.169252 14365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:51.170328 14365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:51.171556 14365 hybrid_clock.cc:648] HybridClock initialized: now 1781922711171479 us; error 68 us; skew 500 ppm
I20260620 02:31:51.171873 14365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:51 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:51.175379 14365 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:31:51.177147 14365 webserver.cc:460] Webserver started at http://127.0.97.1:46265/ using document root <none> and password file <none>
I20260620 02:31:51.177924 14365 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:51.178134 14365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:51.178555 14365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:51.181871 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "92a561d764cb499b9980263ac9ebbede"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "413904337c9e2c005d074e487451a7e6"
server_key_iv: "4df11c87e9a7596be7d3a40f6051b325"
server_key_version: "encryptionkey@0"
I20260620 02:31:51.182703 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "92a561d764cb499b9980263ac9ebbede"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "413904337c9e2c005d074e487451a7e6"
server_key_iv: "4df11c87e9a7596be7d3a40f6051b325"
server_key_version: "encryptionkey@0"
I20260620 02:31:51.188194 14365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:31:51.193177 14380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:51.195228 14365 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:31:51.195513 14365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "92a561d764cb499b9980263ac9ebbede"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "413904337c9e2c005d074e487451a7e6"
server_key_iv: "4df11c87e9a7596be7d3a40f6051b325"
server_key_version: "encryptionkey@0"
I20260620 02:31:51.195765 14365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:51.214455 14365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:51.219034 14365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:51.219353 14365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:51.220371 14365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:51.221868 14365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:51.221976 14365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:51.222132 14365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:51.222206 14365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:51.257941 14365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44569
I20260620 02:31:51.257951 14492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44569 every 8 connection(s)
I20260620 02:31:51.259478 14365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:31:51.267853 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14365
I20260620 02:31:51.268107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:31:51.268810 388 external_mini_cluster.cc:1428] Setting key 6b132e1956b4062a772d64625e7b8dcc
Jun 20 02:31:51 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:51.282164 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:51.262096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36009 (local address 127.0.97.62:44705)
0620 02:31:51.262571 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:51.262576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:51.263528 (+ 952us) server_negotiation.cc:407] Connection header received
0620 02:31:51.265496 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:51.265502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:51.265597 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:51.265723 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:51.267284 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.268104 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.269843 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.270309 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.270460 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:51.274334 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:51.274344 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:51.274351 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:51.274402 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:51.278280 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:51.279284 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:51.279292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:51.279299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:51.279481 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:51.280048 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:51.280054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:51.280059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:51.280421 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:51.280614 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:51.281481 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 02:31:51.281744 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260620 02:31:51.283304 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:51.262604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:36009)
0620 02:31:51.263324 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:51.263380 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:51.265207 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:51.265972 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:51.265981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:51.266508 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:51.267092 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:51.267110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.268283 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.268288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:51.269684 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:51.269697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.270556 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.270565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:51.270814 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:51.271530 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:51.271559 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:51.274058 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:51.278451 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:51.278463 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:51.278479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:51.279105 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:51.279608 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:51.279611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:51.279614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:51.279900 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:51.280580 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:51.280589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:51.280989 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 02:31:51.281370 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 02:31:51.281989 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":297,"threads_started":1}
I20260620 02:31:51.285081 14493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44705
I20260620 02:31:51.285424 14493 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:51.286408 14493 heartbeater.cc:511] Master 127.0.97.62:44705 requested a full tablet report, sending...
I20260620 02:31:51.288762 14303 ts_manager.cc:194] Registered new tserver with Master: 92a561d764cb499b9980263ac9ebbede (127.0.97.1:44569)
I20260620 02:31:51.290302 14303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36009
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:31:51.328830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44705
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:51.667369 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:51.667680 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:51.667776 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:51.675271 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:51.675407 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:51.675526 14499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:31:51.685791 14499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:51.687108 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:51.688387 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:51.702754 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:51.702939 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:51.703197 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:51.703980 14499 server_base.cc:1034] running on GCE node
I20260620 02:31:51.704806 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:51.705932 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:51.707209 14499 hybrid_clock.cc:648] HybridClock initialized: now 1781922711707154 us; error 57 us; skew 500 ppm
I20260620 02:31:51.707614 14499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:51 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:51.711670 14499 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:31:51.713446 14499 webserver.cc:460] Webserver started at http://127.0.97.2:33467/ using document root <none> and password file <none>
I20260620 02:31:51.714129 14499 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:51.714234 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:51.714501 14499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:51.717517 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "87f10482a2e041e28791219089d2f0fb"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "17d3724078e28482b9b1969fe859943d"
server_key_iv: "0caa9a333875c0796bab8d0b4eb95e48"
server_key_version: "encryptionkey@0"
I20260620 02:31:51.718304 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "87f10482a2e041e28791219089d2f0fb"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "17d3724078e28482b9b1969fe859943d"
server_key_iv: "0caa9a333875c0796bab8d0b4eb95e48"
server_key_version: "encryptionkey@0"
I20260620 02:31:51.723543 14499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:31:51.727272 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:51.728816 14499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:31:51.729049 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "87f10482a2e041e28791219089d2f0fb"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "17d3724078e28482b9b1969fe859943d"
server_key_iv: "0caa9a333875c0796bab8d0b4eb95e48"
server_key_version: "encryptionkey@0"
I20260620 02:31:51.729249 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:51.747983 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:51.753499 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:51.753777 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:51.754889 14499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:51.756321 14499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:51.756423 14499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:51.756521 14499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:51.756567 14499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:51.792444 14499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41215
I20260620 02:31:51.792539 14626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41215 every 8 connection(s)
I20260620 02:31:51.793875 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:31:51.795140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14499
I20260620 02:31:51.795356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:31:51.795811 388 external_mini_cluster.cc:1428] Setting key 3df9586a52c8aea8939bbcb5c273be17
Jun 20 02:31:51 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:51.817260 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:51.796801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47557 (local address 127.0.97.62:44705)
0620 02:31:51.797335 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:51.797342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:51.798440 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 02:31:51.800541 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:51.800549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:51.800657 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:51.800773 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:51.802543 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.803841 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.805570 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.806307 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.806473 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:51.810879 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:51.810884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:51.810888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:51.810924 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:51.814001 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:51.814814 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:51.814819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:51.814821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:51.814949 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:51.815406 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:51.815410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:51.815412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:51.815697 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:51.815868 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:51.816598 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 02:31:51.816842 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260620 02:31:51.818356 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:51.797411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:47557)
0620 02:31:51.798270 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:51.798318 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:51.800199 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:51.800983 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:51.800993 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:51.801603 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:51.802337 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:51.802362 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.804044 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.804048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:51.805417 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:51.805431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:51.806546 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:51.806551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:51.806896 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:51.807727 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:51.807773 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:51.810601 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:51.814162 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:51.814172 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:51.814190 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:51.814653 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:51.815076 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:51.815081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:51.815085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:51.815280 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:51.815825 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:51.815833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:51.816195 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 02:31:51.816500 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 02:31:51.817078 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":364,"threads_started":1}
I20260620 02:31:51.820041 14627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44705
I20260620 02:31:51.820399 14627 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:51.821410 14627 heartbeater.cc:511] Master 127.0.97.62:44705 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:31:51.823091 14303 ts_manager.cc:194] Registered new tserver with Master: 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215)
I20260620 02:31:51.823992 14303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:31:51.856698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44705
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:52.198393 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:52.198766 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:52.198939 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:52.206482 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:52.206655 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:52.206828 14633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:31:52.217715 14633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:52.219092 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:52.220435 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:52.234225 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:52.234203 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:52.234179 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:52.235496 14633 server_base.cc:1034] running on GCE node
I20260620 02:31:52.236119 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:52.237130 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:52.238356 14633 hybrid_clock.cc:648] HybridClock initialized: now 1781922712238311 us; error 55 us; skew 500 ppm
I20260620 02:31:52.238677 14633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:52 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:52.242677 14633 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:31:52.244993 14633 webserver.cc:460] Webserver started at http://127.0.97.3:38113/ using document root <none> and password file <none>
I20260620 02:31:52.246215 14633 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:52.246380 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:52.246970 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:31:52.251873 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "0547e0c9d6644d158c773b75a26bdcff"
format_stamp: "Formatted at 2026-06-20 02:31:52 on dist-test-slave-r706"
server_key: "78e28ac9eebf79062fc6780a3400feb6"
server_key_iv: "94f94d588fdd3747afcc2f0222542737"
server_key_version: "encryptionkey@0"
I20260620 02:31:52.253181 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0547e0c9d6644d158c773b75a26bdcff"
format_stamp: "Formatted at 2026-06-20 02:31:52 on dist-test-slave-r706"
server_key: "78e28ac9eebf79062fc6780a3400feb6"
server_key_iv: "94f94d588fdd3747afcc2f0222542737"
server_key_version: "encryptionkey@0"
I20260620 02:31:52.262157 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 02:31:52.267248 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:52.269551 14633 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:31:52.269790 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "0547e0c9d6644d158c773b75a26bdcff"
format_stamp: "Formatted at 2026-06-20 02:31:52 on dist-test-slave-r706"
server_key: "78e28ac9eebf79062fc6780a3400feb6"
server_key_iv: "94f94d588fdd3747afcc2f0222542737"
server_key_version: "encryptionkey@0"
I20260620 02:31:52.270147 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:52.294303 14493 heartbeater.cc:503] Master 127.0.97.62:44705 was elected leader, sending a full tablet report...
I20260620 02:31:52.295084 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:52.300565 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:52.300927 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:52.302130 14633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:52.303593 14633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:52.303673 14633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:52.303858 14633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:52.303915 14633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:52.341459 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39031
I20260620 02:31:52.341512 14760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39031 every 8 connection(s)
I20260620 02:31:52.342916 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:31:52.344858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14633
I20260620 02:31:52.345077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:31:52.345688 388 external_mini_cluster.cc:1428] Setting key 52c8a0e3c495532c05ec52201e2ad49c
Jun 20 02:31:52 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:52.365690 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.345926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51845 (local address 127.0.97.62:44705)
0620 02:31:52.346458 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.346466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.347598 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 02:31:52.349904 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.349910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.350006 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.350125 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:52.351877 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.353224 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.354802 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.355224 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.355366 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.358923 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:52.358928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:52.358931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:52.358987 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:52.362176 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:52.363021 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:52.363027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:52.363033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:52.363209 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:52.363669 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:52.363673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:52.363676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:52.364061 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:52.364267 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.365224 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.365417 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":243,"threads_started":1}
I20260620 02:31:52.366766 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.346529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:51845)
0620 02:31:52.347445 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:52.347482 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:52.349582 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:52.350378 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:52.350387 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:52.350900 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:52.351466 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.351483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.353387 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.353393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.354649 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.354661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.355466 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.355473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.355708 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.356372 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:52.356404 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:52.358710 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:52.362321 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:52.362328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:52.362343 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:52.362801 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:52.363342 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:52.363345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:52.363348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:52.363537 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:52.364290 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:52.364297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:52.364748 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 02:31:52.365126 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 02:31:52.365581 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":336,"threads_started":1}
I20260620 02:31:52.368240 14761 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44705
I20260620 02:31:52.368525 14761 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:52.369304 14761 heartbeater.cc:511] Master 127.0.97.62:44705 requested a full tablet report, sending...
I20260620 02:31:52.370733 14303 ts_manager.cc:194] Registered new tserver with Master: 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031)
I20260620 02:31:52.371543 14303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51845
I20260620 02:31:52.374076 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:31:52.391841 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.379025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.97.62:44705)
0620 02:31:52.379195 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.379199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.379640 (+ 441us) server_negotiation.cc:407] Connection header received
0620 02:31:52.379747 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.379751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.379831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.379927 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:52.381416 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.382166 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.383439 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.383858 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.383952 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.386605 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:52.386610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:52.386613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:52.386646 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:52.389320 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:52.389985 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:52.389989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:52.389992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:52.390108 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:52.390427 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:52.390429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:52.390432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:52.390759 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:52.390963 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.391410 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.391606 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:31:52.396636 14303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:31:52.404430 14303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:31:52.432696 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.417663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.0.1:47660)
0620 02:31:52.418764 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:52.418814 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:52.419047 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:52.420936 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:52.420947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:52.421032 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:52.421579 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.421596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.423829 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.423838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.427460 (+ 3622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.427482 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.431014 (+ 3532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.431019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.431233 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.431302 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 02:31:52.431445 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:31:52.431724 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":1,"thread_start_us":568,"threads_started":1}
I20260620 02:31:52.434453 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.417833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41215 (local address 127.0.0.1:33310)
0620 02:31:52.421753 (+ 3920us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:52.421780 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:52.421935 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:52.423033 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:52.423040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:52.423077 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:52.423606 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.423621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.426568 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.426579 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.429631 (+ 3052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.429652 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.432708 (+ 3056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.432720 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.433606 (+ 886us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.433663 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 02:31:52.433893 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 02:31:52.434109 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1753,"thread_start_us":1608,"threads_started":1}
I20260620 02:31:52.434568 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.418475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.0.97.2:41215)
0620 02:31:52.421656 (+ 3181us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.421669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.422071 (+ 402us) server_negotiation.cc:407] Connection header received
0620 02:31:52.422191 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.422202 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.422502 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.422741 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:52.423847 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.426333 (+ 2486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.429879 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.431879 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.432038 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.432330 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.434064 (+ 1734us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.434266 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3007,"thread_start_us":2892,"threads_started":1}
I20260620 02:31:52.434901 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.417924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.0.97.3:39031)
0620 02:31:52.420151 (+ 2227us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.420164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.420206 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:31:52.420332 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.420343 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.420666 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.420910 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:52.421881 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.423630 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.428860 (+ 5230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.430777 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.434037 (+ 3260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.434274 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.434348 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.434522 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2029,"thread_start_us":1860,"threads_started":1}
I20260620 02:31:52.438222 14562 tablet_service.cc:1467] Processing CreateTablet for tablet cb9210a93250486da7cd1225741ae71c (DEFAULT_TABLE table=test-table [id=76781e16e35c42a7bec36d6dc631f47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:52.438218 14696 tablet_service.cc:1467] Processing CreateTablet for tablet cb9210a93250486da7cd1225741ae71c (DEFAULT_TABLE table=test-table [id=76781e16e35c42a7bec36d6dc631f47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:52.439394 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9210a93250486da7cd1225741ae71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:52.439508 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9210a93250486da7cd1225741ae71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:52.445060 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.419830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35130)
0620 02:31:52.433801 (+ 13971us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:52.433823 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:52.433962 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:52.434784 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:52.434788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:52.434814 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:52.435041 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.435052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.440370 (+ 5318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.440376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.442298 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.442323 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.444117 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.444124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.444313 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.444393 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 02:31:52.444508 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:31:52.444667 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13836,"thread_start_us":13922,"threads_started":1}
I20260620 02:31:52.445107 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.420177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.0.97.1:44569)
0620 02:31:52.433581 (+ 13404us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.433591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.434106 (+ 515us) server_negotiation.cc:407] Connection header received
0620 02:31:52.434211 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.434221 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.434560 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.435250 (+ 690us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:52.435522 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.440201 (+ 4679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.442490 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.443939 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.444079 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.444293 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.444568 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.444741 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13221,"thread_start_us":13052,"threads_started":1}
I20260620 02:31:52.450678 14428 tablet_service.cc:1467] Processing CreateTablet for tablet cb9210a93250486da7cd1225741ae71c (DEFAULT_TABLE table=test-table [id=76781e16e35c42a7bec36d6dc631f47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:52.452235 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9210a93250486da7cd1225741ae71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:52.453214 14780 tablet_bootstrap.cc:492] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Bootstrap starting.
I20260620 02:31:52.455284 14781 tablet_bootstrap.cc:492] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Bootstrap starting.
I20260620 02:31:52.457370 14780 tablet_bootstrap.cc:654] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:52.458777 14780 log.cc:824] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:52.459435 14781 tablet_bootstrap.cc:654] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:52.460711 14781 log.cc:824] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:52.462215 14780 tablet_bootstrap.cc:492] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: No bootstrap required, opened a new log
I20260620 02:31:52.462512 14780 ts_tablet_manager.cc:1397] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 02:31:52.463538 14781 tablet_bootstrap.cc:492] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: No bootstrap required, opened a new log
I20260620 02:31:52.463809 14781 ts_tablet_manager.cc:1397] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:31:52.466583 14784 tablet_bootstrap.cc:492] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Bootstrap starting.
I20260620 02:31:52.470403 14784 tablet_bootstrap.cc:654] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:52.471455 14780 raft_consensus.cc:348] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.471772 14784 log.cc:824] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:52.472076 14780 raft_consensus.cc:374] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:52.472242 14780 raft_consensus.cc:729] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f10482a2e041e28791219089d2f0fb, State: Initialized, Role: FOLLOWER
I20260620 02:31:52.473066 14780 consensus_queue.cc:260] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.473903 14781 raft_consensus.cc:348] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.475656 14784 tablet_bootstrap.cc:492] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: No bootstrap required, opened a new log
I20260620 02:31:52.484238 14784 ts_tablet_manager.cc:1397] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260620 02:31:52.491532 14781 raft_consensus.cc:374] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:52.491732 14781 raft_consensus.cc:729] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0547e0c9d6644d158c773b75a26bdcff, State: Initialized, Role: FOLLOWER
I20260620 02:31:52.492761 14781 consensus_queue.cc:260] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.501106 14784 raft_consensus.cc:348] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.501811 14784 raft_consensus.cc:374] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:52.502002 14784 raft_consensus.cc:729] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a561d764cb499b9980263ac9ebbede, State: Initialized, Role: FOLLOWER
I20260620 02:31:52.503021 14784 consensus_queue.cc:260] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.505381 14627 heartbeater.cc:503] Master 127.0.97.62:44705 was elected leader, sending a full tablet report...
I20260620 02:31:52.511334 14761 heartbeater.cc:503] Master 127.0.97.62:44705 was elected leader, sending a full tablet report...
I20260620 02:31:52.512621 14781 ts_tablet_manager.cc:1428] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Time spent starting tablet: real 0.049s user 0.021s sys 0.007s
I20260620 02:31:52.514467 14780 ts_tablet_manager.cc:1428] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Time spent starting tablet: real 0.052s user 0.016s sys 0.010s
I20260620 02:31:52.527905 14784 ts_tablet_manager.cc:1428] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Time spent starting tablet: real 0.038s user 0.025s sys 0.010s
W20260620 02:31:52.546162 14628 tablet.cc:2369] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:52.573246 14788 raft_consensus.cc:482] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:31:52.573638 14788 raft_consensus.cc:504] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.579286 14788 leader_election.cc:288] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031), 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215)
I20260620 02:31:52.593127 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.577022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59373 (local address 127.0.97.3:39031)
0620 02:31:52.577272 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.577281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.580674 (+ 3393us) server_negotiation.cc:407] Connection header received
0620 02:31:52.581098 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.581107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.581218 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.581306 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:52.584238 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.585616 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.589945 (+ 4329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.591543 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.591669 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.591802 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.592537 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.592728 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:31:52.594167 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.576434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.97.1:59373)
0620 02:31:52.580503 (+ 4069us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:52.580531 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:52.580855 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:52.583184 (+ 2329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:52.583192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:52.583234 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:52.584015 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.584034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.585842 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.585851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.589691 (+ 3840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.589711 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.592052 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.592061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.592310 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.592358 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 02:31:52.593440 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0620 02:31:52.593624 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3884,"mutex_wait_us":4,"thread_start_us":3257,"threads_started":1}
W20260620 02:31:52.595122 14762 tablet.cc:2369] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:52.596503 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.576800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35437 (local address 127.0.97.2:41215)
0620 02:31:52.577049 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.577057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.587012 (+ 9955us) server_negotiation.cc:407] Connection header received
0620 02:31:52.587238 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.587248 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.587351 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.587454 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:52.588467 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.589952 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.593322 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.594983 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.595143 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.595266 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.595958 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.596161 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:31:52.596573 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9210a93250486da7cd1225741ae71c" candidate_uuid: "92a561d764cb499b9980263ac9ebbede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0547e0c9d6644d158c773b75a26bdcff" is_pre_election: true
I20260620 02:31:52.597259 14716 raft_consensus.cc:2454] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a561d764cb499b9980263ac9ebbede in term 0.
I20260620 02:31:52.597424 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.577397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41215 (local address 127.0.97.1:35437)
0620 02:31:52.586874 (+ 9477us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:52.586900 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:52.587058 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:52.587647 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:52.587655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:52.587692 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:52.588249 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.588265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.590195 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.590203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.593113 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:52.593134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.595492 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.595500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:52.595732 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.595780 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 02:31:52.596765 (+ 985us) client_negotiation.cc:240] Negotiation successful
0620 02:31:52.596943 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9312,"thread_start_us":9126,"threads_started":1}
I20260620 02:31:52.598203 14381 leader_election.cc:302] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0547e0c9d6644d158c773b75a26bdcff, 92a561d764cb499b9980263ac9ebbede; no voters:
I20260620 02:31:52.598755 14788 raft_consensus.cc:2781] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:31:52.598639 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9210a93250486da7cd1225741ae71c" candidate_uuid: "92a561d764cb499b9980263ac9ebbede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f10482a2e041e28791219089d2f0fb" is_pre_election: true
I20260620 02:31:52.598930 14788 raft_consensus.cc:482] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:31:52.599057 14788 raft_consensus.cc:3037] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:52.599293 14582 raft_consensus.cc:2454] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a561d764cb499b9980263ac9ebbede in term 0.
I20260620 02:31:52.603480 14788 raft_consensus.cc:504] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.604956 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9210a93250486da7cd1225741ae71c" candidate_uuid: "92a561d764cb499b9980263ac9ebbede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f10482a2e041e28791219089d2f0fb"
I20260620 02:31:52.604950 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9210a93250486da7cd1225741ae71c" candidate_uuid: "92a561d764cb499b9980263ac9ebbede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0547e0c9d6644d158c773b75a26bdcff"
I20260620 02:31:52.605249 14582 raft_consensus.cc:3037] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:52.605278 14716 raft_consensus.cc:3037] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:52.606002 14788 leader_election.cc:288] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 1 election: Requested vote from peers 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031), 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215)
I20260620 02:31:52.609555 14582 raft_consensus.cc:2454] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a561d764cb499b9980263ac9ebbede in term 1.
I20260620 02:31:52.609699 14716 raft_consensus.cc:2454] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a561d764cb499b9980263ac9ebbede in term 1.
I20260620 02:31:52.610193 14384 leader_election.cc:302] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87f10482a2e041e28791219089d2f0fb, 92a561d764cb499b9980263ac9ebbede; no voters:
I20260620 02:31:52.610735 14788 raft_consensus.cc:2781] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:52.611370 14788 raft_consensus.cc:686] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 1 LEADER]: Becoming Leader. State: Replica: 92a561d764cb499b9980263ac9ebbede, State: Running, Role: LEADER
I20260620 02:31:52.612099 14788 consensus_queue.cc:237] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:52.618114 14301 catalog_manager.cc:5495] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede reported cstate change: term changed from 0 to 1, leader changed from <none> to 92a561d764cb499b9980263ac9ebbede (127.0.97.1). New cstate: current_term: 1 leader_uuid: "92a561d764cb499b9980263ac9ebbede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } health_report { overall_health: HEALTHY } } }
I20260620 02:31:52.685886 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.676942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.97.1:44569)
0620 02:31:52.677258 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.677267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.677295 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:31:52.677582 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.677601 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.677739 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.677925 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:31:52.678500 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.679849 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.682423 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.683284 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.683438 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.684239 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:31:52.685070 (+ 831us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:31:52.685241 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.685383 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.685497 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 02:31:52.698251 14582 raft_consensus.cc:1261] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Refusing update from remote peer 92a561d764cb499b9980263ac9ebbede: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:52.698490 14716 raft_consensus.cc:1261] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Refusing update from remote peer 92a561d764cb499b9980263ac9ebbede: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:52.699555 14791 consensus_queue.cc:1035] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:31:52.700968 14791 consensus_queue.cc:1035] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [LEADER]: Connected to new peer: Peer: permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:31:52.739892 14794 mvcc.cc:205] Tried to move back new op lower bound from 7298755431198310400 to 7298755430865104896. Current Snapshot: MvccSnapshot[applied={T|T < 7298755431198310400}]
I20260620 02:31:52.759919 14798 mvcc.cc:205] Tried to move back new op lower bound from 7298755431198310400 to 7298755430865104896. Current Snapshot: MvccSnapshot[applied={T|T < 7298755431198310400 or (T in {7298755431198310400})}]
I20260620 02:31:52.776786 14301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:31:52.777383 14301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:31:52.783756 14301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Sending DeleteTablet for 3 replicas of tablet cb9210a93250486da7cd1225741ae71c
I20260620 02:31:52.785380 14696 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9210a93250486da7cd1225741ae71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47660
I20260620 02:31:52.789258 14562 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9210a93250486da7cd1225741ae71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33310
I20260620 02:31:52.789985 14804 tablet_replica.cc:330] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: stopping tablet replica
I20260620 02:31:52.790477 14804 raft_consensus.cc:2229] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:31:52.791059 14804 raft_consensus.cc:2258] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:52.792542 14805 tablet_replica.cc:330] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: stopping tablet replica
I20260620 02:31:52.793519 14428 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9210a93250486da7cd1225741ae71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35130
I20260620 02:31:52.794478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14365
I20260620 02:31:52.805899 14806 tablet_replica.cc:330] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: stopping tablet replica
I20260620 02:31:52.806500 14806 raft_consensus.cc:2229] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:31:52.807325 14806 raft_consensus.cc:2258] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:52.809893 14806 ts_tablet_manager.cc:1905] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:52.820449 14804 ts_tablet_manager.cc:1905] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:52.821138 14805 raft_consensus.cc:2229] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:31:52.822077 14805 raft_consensus.cc:2258] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:52.824472 14805 ts_tablet_manager.cc:1905] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:31:52 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:31:52.838608 14806 ts_tablet_manager.cc:1918] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:52.838819 14806 log.cc:1196] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/cb9210a93250486da7cd1225741ae71c
I20260620 02:31:52.839453 14806 ts_tablet_manager.cc:1939] T cb9210a93250486da7cd1225741ae71c P 92a561d764cb499b9980263ac9ebbede: Deleting consensus metadata
I20260620 02:31:52.845930 14804 ts_tablet_manager.cc:1918] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:52.846160 14804 log.cc:1196] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/cb9210a93250486da7cd1225741ae71c
I20260620 02:31:52.848613 14805 ts_tablet_manager.cc:1918] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:52.848922 14805 log.cc:1196] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/cb9210a93250486da7cd1225741ae71c
I20260620 02:31:52.849668 14805 ts_tablet_manager.cc:1939] T cb9210a93250486da7cd1225741ae71c P 87f10482a2e041e28791219089d2f0fb: Deleting consensus metadata
I20260620 02:31:52.852209 14290 catalog_manager.cc:4841] TS 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215): tablet cb9210a93250486da7cd1225741ae71c (table test-table [id=76781e16e35c42a7bec36d6dc631f47e]) successfully deleted
I20260620 02:31:52.856261 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:52.795982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.0.97.1:44569)
0620 02:31:52.796204 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:52.796213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:52.813697 (+ 17484us) server_negotiation.cc:407] Connection header received
0620 02:31:52.814019 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:52.814025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:52.814122 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:52.814209 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:52.816320 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.817534 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.822852 (+ 5318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:52.823587 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:52.823695 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:52.846645 (+ 22950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:52.846673 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:52.846696 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:52.846755 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:52.851585 (+ 4830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:52.852716 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:52.852726 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:52.852731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:52.852926 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:52.853485 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:52.853490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:52.853493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:52.853972 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:52.854299 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:52.855588 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0620 02:31:52.855849 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:31:52.857091 14488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:52.858705 14804 ts_tablet_manager.cc:1939] T cb9210a93250486da7cd1225741ae71c P 0547e0c9d6644d158c773b75a26bdcff: Deleting consensus metadata
I20260620 02:31:52.861765 14287 catalog_manager.cc:4841] TS 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031): tablet cb9210a93250486da7cd1225741ae71c (table test-table [id=76781e16e35c42a7bec36d6dc631f47e]) successfully deleted
I20260620 02:31:52.862308 14290 catalog_manager.cc:4841] TS 92a561d764cb499b9980263ac9ebbede (127.0.97.1:44569): tablet cb9210a93250486da7cd1225741ae71c (table test-table [id=76781e16e35c42a7bec36d6dc631f47e]) successfully deleted
I20260620 02:31:53.011310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14365
I20260620 02:31:53.033118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14499
Jun 20 02:31:53 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:31:53.049402 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:53.033879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.0.97.2:41215)
0620 02:31:53.034064 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:53.034073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:53.034098 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:31:53.034333 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:53.034339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:53.034425 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:53.034540 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:53.035837 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:53.036539 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:53.038157 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:53.038566 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:53.038713 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:53.043223 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:53.043238 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:53.043253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:53.043288 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:53.046375 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:53.047076 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:53.047083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:53.047086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:53.047279 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:53.047718 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:53.047725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:53.047729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:53.048113 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:53.048423 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:53.048752 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 02:31:53.048983 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:31:53.050096 14622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:53.140904 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14499
I20260620 02:31:53.159754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14633
Jun 20 02:31:53 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:31:53.175836 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:53.160599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.0.97.3:39031)
0620 02:31:53.161142 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:53.161147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:53.161166 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:31:53.161221 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:53.161225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:53.161288 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:53.161394 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:53.162866 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:53.163618 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:53.164987 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:53.165501 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:53.165603 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:53.170143 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:53.170160 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:53.170176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:53.170211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:53.172847 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:53.173644 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:53.173652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:53.173654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:53.173808 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:53.174240 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:53.174244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:53.174246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:53.174626 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:53.174849 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:53.175203 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:31:53.175380 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260620 02:31:53.176541 14756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:53.265296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14633
I20260620 02:31:53.282913 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14271
I20260620 02:31:53.284034 14333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:53.396260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14271
I20260620 02:31:53.420552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44705
--webserver_interface=127.0.97.62
--webserver_port=35089
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:53.762224 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:53.762529 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:53.762717 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:53.770291 14813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:53.770416 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:53.770503 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:53.770609 14813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:53.770695 14813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:53.781150 14813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:53.782526 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:53.783864 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:53.809775 14820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:53.809753 14822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:53.809823 14819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:53.816640 14813 server_base.cc:1034] running on GCE node
I20260620 02:31:53.817708 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:53.819473 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:53.820698 14813 hybrid_clock.cc:648] HybridClock initialized: now 1781922713820644 us; error 40 us; skew 500 ppm
I20260620 02:31:53.821205 14813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:53 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:53.827848 14813 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:53.829825 14813 webserver.cc:460] Webserver started at http://127.0.97.62:35089/ using document root <none> and password file <none>
I20260620 02:31:53.830648 14813 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:53.830816 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:53.836891 14813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:31:53.839932 14829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:53.841338 14813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:31:53.841534 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "1485a16502cb49949e9518de3112c0a2"
format_stamp: "Formatted at 2026-06-20 02:31:50 on dist-test-slave-r706"
server_key: "7141c377b3b2ca43de3410aace0a1999"
server_key_iv: "86519d818ad34657fe9e7aaaeb5009e2"
server_key_version: "encryptionkey@0"
I20260620 02:31:53.842648 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:53.870527 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:53.874963 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:53.875305 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:53.895273 14813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44705
I20260620 02:31:53.895298 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44705 every 8 connection(s)
I20260620 02:31:53.896855 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:53.903946 14881 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:31:53.906836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14813
I20260620 02:31:53.907431 14881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap starting.
I20260620 02:31:53.907729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44569
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44705
--webserver_port=46265
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:31:53.929606 14881 log.cc:824] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:53.942322 14881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:31:53.942924 14881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap complete.
I20260620 02:31:53.952699 14881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:53.953738 14881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1485a16502cb49949e9518de3112c0a2, State: Initialized, Role: FOLLOWER
I20260620 02:31:53.954380 14881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:53.954633 14881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:31:53.954757 14881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:31:53.954958 14881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:31:53.958968 14881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:53.959545 14881 leader_election.cc:302] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1485a16502cb49949e9518de3112c0a2; no voters:
I20260620 02:31:53.960222 14881 leader_election.cc:288] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:31:53.960357 14885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:31:53.961691 14885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 LEADER]: Becoming Leader. State: Replica: 1485a16502cb49949e9518de3112c0a2, State: Running, Role: LEADER
I20260620 02:31:53.962258 14885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:53.962738 14881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:53.965173 14886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1485a16502cb49949e9518de3112c0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } } }
I20260620 02:31:53.965548 14886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:53.964959 14887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1485a16502cb49949e9518de3112c0a2. Latest consensus state: current_term: 2 leader_uuid: "1485a16502cb49949e9518de3112c0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } } }
I20260620 02:31:53.966321 14887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:53.974196 14890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:53.985286 14890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:53.987244 14890 catalog_manager.cc:1259] Loaded cluster ID: 84c7ba71dd894e85a4e00e7d6b57ff49
I20260620 02:31:53.987387 14890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:31:53.991559 14890 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 02:31:54.017604 14890 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 02:31:54.029593 14890 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:31:54.030391 14902 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:31:54.031649 14890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Loaded TSK: 0
I20260620 02:31:54.032805 14890 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:31:54.285511 14883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:54.285799 14883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:54.285950 14883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:54.293756 14883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:54.293874 14883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:54.294051 14883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:31:54.304283 14883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46265
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:54.305670 14883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:54.307011 14883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:54.320767 14909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:54.321043 14908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:54.321156 14911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:54.323390 14883 server_base.cc:1034] running on GCE node
I20260620 02:31:54.324168 14883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:54.325377 14883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:54.326637 14883 hybrid_clock.cc:648] HybridClock initialized: now 1781922714326575 us; error 49 us; skew 500 ppm
I20260620 02:31:54.327005 14883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:54 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:54.330367 14883 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:31:54.332057 14883 webserver.cc:460] Webserver started at http://127.0.97.1:46265/ using document root <none> and password file <none>
I20260620 02:31:54.332808 14883 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:54.332971 14883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:54.338760 14883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 02:31:54.341689 14918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:54.343134 14883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:31:54.343321 14883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "92a561d764cb499b9980263ac9ebbede"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "413904337c9e2c005d074e487451a7e6"
server_key_iv: "4df11c87e9a7596be7d3a40f6051b325"
server_key_version: "encryptionkey@0"
I20260620 02:31:54.344309 14883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:54.369529 14883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:54.374023 14883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:54.374366 14883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:54.375439 14883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:54.376895 14883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:54.376986 14883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:54.377202 14883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:54.377302 14883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:54.411064 14883 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44569
I20260620 02:31:54.411171 15030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44569 every 8 connection(s)
I20260620 02:31:54.412516 14883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:31:54.415660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 14883
I20260620 02:31:54.416493 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41215
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44705
--webserver_port=33467
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:31:54 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:54.438133 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:54.415711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:41239)
0620 02:31:54.416468 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:54.416510 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:54.418354 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:54.419728 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:54.419740 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:54.420354 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:54.420877 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:54.420893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.422540 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.422551 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:54.424204 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:54.424223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.425305 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.425315 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:54.425631 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:54.426356 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:54.426390 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:54.429132 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:54.433988 (+ 4856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:54.433995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:54.434006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:54.434408 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:54.434937 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:54.434941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:54.434944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:54.435124 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:54.436094 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:54.436100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:54.436393 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 02:31:54.436636 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 02:31:54.437121 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":337,"threads_started":1}
I20260620 02:31:54.438807 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:54.415383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41239 (local address 127.0.97.62:44705)
0620 02:31:54.416068 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:54.416095 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:54.416854 (+ 759us) server_negotiation.cc:407] Connection header received
0620 02:31:54.418894 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:54.418905 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:54.419225 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:54.419582 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:54.421129 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.422362 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.424373 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.425069 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.425218 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:54.429406 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:54.429419 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:54.429440 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:54.429490 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:54.433814 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:54.434548 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:54.434559 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:54.434569 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:54.434787 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:54.435245 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:54.435253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:54.435259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:54.435945 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:54.436174 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:54.436927 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 02:31:54.437466 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":384,"threads_started":1}
I20260620 02:31:54.441596 15031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44705
I20260620 02:31:54.441953 15031 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:54.442910 15031 heartbeater.cc:511] Master 127.0.97.62:44705 requested a full tablet report, sending...
I20260620 02:31:54.445461 14846 ts_manager.cc:194] Registered new tserver with Master: 92a561d764cb499b9980263ac9ebbede (127.0.97.1:44569)
I20260620 02:31:54.447036 14846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41239
W20260620 02:31:54.750110 15035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:54.750399 15035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:54.750577 15035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:54.757973 15035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:54.758081 15035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:54.758275 15035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:31:54.768241 15035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33467
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:54.769591 15035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:54.770980 15035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:54.784379 15040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:54.784492 15041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:54.784561 15043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:54.785593 15035 server_base.cc:1034] running on GCE node
I20260620 02:31:54.786218 15035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:54.787230 15035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:54.788463 15035 hybrid_clock.cc:648] HybridClock initialized: now 1781922714788399 us; error 61 us; skew 500 ppm
I20260620 02:31:54.788790 15035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:54 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:54.792147 15035 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:31:54.793829 15035 webserver.cc:460] Webserver started at http://127.0.97.2:33467/ using document root <none> and password file <none>
I20260620 02:31:54.794561 15035 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:54.794734 15035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:54.800468 15035 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:31:54.803522 15050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:54.805341 15035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:31:54.805579 15035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "87f10482a2e041e28791219089d2f0fb"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "17d3724078e28482b9b1969fe859943d"
server_key_iv: "0caa9a333875c0796bab8d0b4eb95e48"
server_key_version: "encryptionkey@0"
I20260620 02:31:54.806664 15035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:54.830245 15035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:54.835000 15035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:54.835357 15035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:54.836447 15035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:54.838056 15035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:54.838208 15035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:54.838371 15035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:54.838474 15035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:54.876142 15035 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41215
I20260620 02:31:54.876243 15162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41215 every 8 connection(s)
I20260620 02:31:54.877795 15035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:31:54.884542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15035
I20260620 02:31:54.885255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39031
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44705
--webserver_port=38113
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:31:54 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:54.899717 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:54.880579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41173 (local address 127.0.97.62:44705)
0620 02:31:54.880758 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:54.880763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:54.881889 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 02:31:54.883819 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:54.883825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:54.883920 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:54.884047 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:54.885824 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.886521 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.888099 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.888564 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.888706 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:54.892616 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:54.892621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:54.892625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:54.892660 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:54.895827 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:54.896886 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:54.896892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:54.896895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:54.897108 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:54.897637 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:54.897642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:54.897646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:54.898021 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:54.898170 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:54.899113 (+ 943us) server_negotiation.cc:299] Negotiation successful
0620 02:31:54.899365 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:31:54.901067 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:54.880921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:41173)
0620 02:31:54.881711 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:54.881749 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:54.883496 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:54.884299 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:54.884308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:54.884901 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:54.885642 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:54.885662 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.886701 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.886708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:54.887941 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:54.887953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:54.888812 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:54.888819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:54.889161 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:54.889953 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:54.890003 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:54.892359 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:54.896009 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:54.896021 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:54.896042 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:54.896726 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:54.897231 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:54.897239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:54.897246 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:54.897492 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:54.898161 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:54.898172 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:54.898615 (+ 443us) client_negotiation.cc:769] Sending connection context
0620 02:31:54.898962 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 02:31:54.899580 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":276,"threads_started":1}
I20260620 02:31:54.902697 15163 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44705
I20260620 02:31:54.903048 15163 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:54.904103 15163 heartbeater.cc:511] Master 127.0.97.62:44705 requested a full tablet report, sending...
I20260620 02:31:54.905985 14846 ts_manager.cc:194] Registered new tserver with Master: 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215)
I20260620 02:31:54.907119 14846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41173
W20260620 02:31:55.235591 15166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:55.235929 15166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:55.236109 15166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:55.243286 15166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:55.243471 15166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:55.243657 15166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:31:55.253539 15166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38113
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:55.254848 15166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:55.256170 15166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:55.270128 15171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:55.270057 15172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:55.270090 15174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:55.270996 15166 server_base.cc:1034] running on GCE node
I20260620 02:31:55.271718 15166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:55.272746 15166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:55.274117 15166 hybrid_clock.cc:648] HybridClock initialized: now 1781922715274009 us; error 92 us; skew 500 ppm
I20260620 02:31:55.274494 15166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:55 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:55.278014 15166 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:31:55.279626 15166 webserver.cc:460] Webserver started at http://127.0.97.3:38113/ using document root <none> and password file <none>
I20260620 02:31:55.280359 15166 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:55.280530 15166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:55.286607 15166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 02:31:55.290531 15181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:55.292557 15166 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:31:55.292783 15166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "0547e0c9d6644d158c773b75a26bdcff"
format_stamp: "Formatted at 2026-06-20 02:31:52 on dist-test-slave-r706"
server_key: "78e28ac9eebf79062fc6780a3400feb6"
server_key_iv: "94f94d588fdd3747afcc2f0222542737"
server_key_version: "encryptionkey@0"
I20260620 02:31:55.294497 15166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:55.317183 15166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:55.321919 15166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:55.322227 15166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:55.323280 15166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:55.324689 15166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:55.324780 15166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:55.324950 15166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:55.325071 15166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 02:31:55.359148 15166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39031
I20260620 02:31:55.359236 15293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39031 every 8 connection(s)
I20260620 02:31:55.360726 15166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:31:55.364531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15166
Jun 20 02:31:55 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:31:55.381417 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.365552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.97.62:44705)
0620 02:31:55.366774 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.366783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.367109 (+ 326us) server_negotiation.cc:407] Connection header received
0620 02:31:55.367325 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.367335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.367399 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.367528 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:55.369407 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.370176 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.371564 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.372020 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.372160 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.375046 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:55.375051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:55.375054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:55.375085 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:55.378479 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.379240 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.379249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.379252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.379390 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.379836 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.379843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.379851 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.380153 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:55.380448 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.380746 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.380984 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":983,"threads_started":1}
I20260620 02:31:55.383381 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.364227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52051 (local address 127.0.97.62:44705)
0620 02:31:55.365320 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.365330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.366116 (+ 786us) server_negotiation.cc:407] Connection header received
0620 02:31:55.368339 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.368347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.368433 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.368558 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:55.370116 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.370892 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.372578 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.373239 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.373380 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.377143 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:55.377149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:55.377154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:55.377208 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:55.380108 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.380942 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.380948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.380952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.381116 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.381558 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.381565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.381567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.381938 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:55.382085 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.382824 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.383037 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921}
I20260620 02:31:55.384382 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.364908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:52051)
0620 02:31:55.365905 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.365971 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:55.368066 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:55.368762 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:55.368770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:55.369368 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:55.369944 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.369961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.371064 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.371068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.372433 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.372445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.373489 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.373497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.373730 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.374347 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:55.374373 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:55.376901 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:55.380247 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:55.380257 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:55.380274 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:55.380805 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:55.381234 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:55.381239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:55.381243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:55.381436 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:55.382076 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:55.382084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:55.382435 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 02:31:55.382729 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 02:31:55.383280 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":362,"threads_started":1}
I20260620 02:31:55.385782 15294 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44705
I20260620 02:31:55.386147 15294 heartbeater.cc:464] Registering TS with master...
I20260620 02:31:55.386997 15294 heartbeater.cc:511] Master 127.0.97.62:44705 requested a full tablet report, sending...
I20260620 02:31:55.388614 14846 ts_manager.cc:194] Registered new tserver with Master: 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031)
I20260620 02:31:55.389655 14846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52051
I20260620 02:31:55.396462 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:31:55.403856 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.397526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.0.97.62:44705)
0620 02:31:55.397736 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.397741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.398221 (+ 480us) server_negotiation.cc:407] Connection header received
0620 02:31:55.398501 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.398505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.398659 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.398841 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:31:55.399269 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.400020 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.401686 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.402102 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.402194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.402791 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:31:55.403341 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:31:55.403449 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.403598 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.403667 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 02:31:55.404825 14846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:31:55.414153 14846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:31:55.437522 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.427341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.0.97.2:41215)
0620 02:31:55.428853 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.428861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.428890 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:31:55.429044 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.429054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.429316 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.430397 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:55.430671 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.432121 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.435150 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.436213 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.436377 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.436613 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.436912 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.437120 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"spinlock_wait_cycles":19328,"thread_start_us":1333,"threads_started":1}
I20260620 02:31:55.437587 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.427132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41215 (local address 127.0.0.1:33334)
0620 02:31:55.428101 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.428145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:55.428310 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:55.429586 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:55.429594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:55.429637 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:55.430183 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.430196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.432263 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.432269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.434938 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.434951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.436445 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.436452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.436682 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.436720 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:31:55.436874 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:31:55.437114 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":45,"thread_start_us":566,"threads_started":1}
I20260620 02:31:55.438602 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.427384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.0.1:47678)
0620 02:31:55.429849 (+ 2465us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.429872 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:55.429992 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:55.430684 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:55.430690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:55.430721 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:55.431138 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.431149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.433255 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.433262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.435162 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.435177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.437286 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.437292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.437524 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.437561 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:31:55.438027 (+ 466us) client_negotiation.cc:240] Negotiation successful
0620 02:31:55.438166 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2329,"mutex_wait_us":1,"thread_start_us":2179,"threads_started":1}
I20260620 02:31:55.439404 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.427980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.0.97.3:39031)
0620 02:31:55.429349 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.429358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.430137 (+ 779us) server_negotiation.cc:407] Connection header received
0620 02:31:55.430246 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.430253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.430481 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.430671 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:55.431531 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.433125 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.435355 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.436771 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.436890 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.437120 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.438936 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.439098 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":1071,"threads_started":1}
I20260620 02:31:55.439656 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.428543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35146)
0620 02:31:55.430714 (+ 2171us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.430729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:55.430858 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:55.431782 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:55.431787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:55.431811 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:55.432189 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.432201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.434180 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.434186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.436529 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.436545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.438798 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.438804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.438980 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.439013 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:31:55.439107 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:31:55.439237 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2043,"mutex_wait_us":115,"thread_start_us":1905,"threads_started":1}
I20260620 02:31:55.440340 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.429078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.97.1:44569)
0620 02:31:55.430251 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.430262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.430996 (+ 734us) server_negotiation.cc:407] Connection header received
0620 02:31:55.431110 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.431119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.431323 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.431526 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:55.432401 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.434030 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.436727 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.438589 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.438737 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.438963 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.439889 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.440047 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":860,"threads_started":1}
I20260620 02:31:55.441720 15098 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4e8259064f4f2791ad3a1fd928b056 (DEFAULT_TABLE table=test-table [id=4d19b5d0cd784a889157f22f8402292d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:55.443265 15098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4e8259064f4f2791ad3a1fd928b056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:55.442912 15229 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4e8259064f4f2791ad3a1fd928b056 (DEFAULT_TABLE table=test-table [id=4d19b5d0cd784a889157f22f8402292d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:55.443959 15229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4e8259064f4f2791ad3a1fd928b056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:55.443640 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4e8259064f4f2791ad3a1fd928b056 (DEFAULT_TABLE table=test-table [id=4d19b5d0cd784a889157f22f8402292d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:31:55.444901 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4e8259064f4f2791ad3a1fd928b056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:31:55.450731 15031 heartbeater.cc:503] Master 127.0.97.62:44705 was elected leader, sending a full tablet report...
I20260620 02:31:55.456974 15310 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Bootstrap starting.
I20260620 02:31:55.461181 15310 tablet_bootstrap.cc:654] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:55.461297 15311 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Bootstrap starting.
I20260620 02:31:55.462659 15310 log.cc:824] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:55.465480 15311 tablet_bootstrap.cc:654] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:55.467088 15311 log.cc:824] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:55.468551 15312 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Bootstrap starting.
I20260620 02:31:55.471995 15311 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: No bootstrap required, opened a new log
I20260620 02:31:55.472292 15311 ts_tablet_manager.cc:1397] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 02:31:55.472363 15312 tablet_bootstrap.cc:654] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Neither blocks nor log segments found. Creating new log.
I20260620 02:31:55.473426 15310 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: No bootstrap required, opened a new log
I20260620 02:31:55.473641 15312 log.cc:824] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:55.473706 15310 ts_tablet_manager.cc:1397] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260620 02:31:55.476842 15312 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: No bootstrap required, opened a new log
I20260620 02:31:55.477216 15312 ts_tablet_manager.cc:1397] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 02:31:55.482008 15310 raft_consensus.cc:348] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.482427 15310 raft_consensus.cc:374] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:55.482529 15310 raft_consensus.cc:729] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0547e0c9d6644d158c773b75a26bdcff, State: Initialized, Role: FOLLOWER
I20260620 02:31:55.482936 15311 raft_consensus.cc:348] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.483352 15311 raft_consensus.cc:374] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:55.483192 15310 consensus_queue.cc:260] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.483455 15311 raft_consensus.cc:729] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f10482a2e041e28791219089d2f0fb, State: Initialized, Role: FOLLOWER
I20260620 02:31:55.484107 15311 consensus_queue.cc:260] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.484388 15294 heartbeater.cc:503] Master 127.0.97.62:44705 was elected leader, sending a full tablet report...
I20260620 02:31:55.485118 15310 ts_tablet_manager.cc:1428] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260620 02:31:55.486164 15163 heartbeater.cc:503] Master 127.0.97.62:44705 was elected leader, sending a full tablet report...
I20260620 02:31:55.486867 15311 ts_tablet_manager.cc:1428] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 02:31:55.489418 15312 raft_consensus.cc:348] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.489909 15312 raft_consensus.cc:374] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:31:55.490015 15312 raft_consensus.cc:729] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a561d764cb499b9980263ac9ebbede, State: Initialized, Role: FOLLOWER
I20260620 02:31:55.490746 15312 consensus_queue.cc:260] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.492484 15312 ts_tablet_manager.cc:1428] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 02:31:55.540773 15316 raft_consensus.cc:482] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:31:55.541148 15316 raft_consensus.cc:504] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.542899 15316 leader_election.cc:288] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215), 92a561d764cb499b9980263ac9ebbede (127.0.97.1:44569)
I20260620 02:31:55.549863 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.543294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41215 (local address 127.0.97.3:34213)
0620 02:31:55.543566 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.543588 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:55.543755 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:55.544334 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:55.544338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:55.544371 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:55.544585 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.544600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.545849 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.545853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.547829 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.547844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.549210 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.549215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.549445 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.549481 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:31:55.549556 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:31:55.549650 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":27}
I20260620 02:31:55.550015 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.543338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34213 (local address 127.0.97.2:41215)
0620 02:31:55.543568 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.543576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.543701 (+ 125us) server_negotiation.cc:407] Connection header received
0620 02:31:55.543927 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.543937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.544016 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.544141 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:55.544757 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.545721 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.547989 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.549064 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.549200 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.549297 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.549614 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.549752 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:31:55.551100 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.544012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39329 (local address 127.0.97.1:44569)
0620 02:31:55.544257 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.544262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.544622 (+ 360us) server_negotiation.cc:407] Connection header received
0620 02:31:55.544780 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.544785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.544869 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.545026 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:31:55.545539 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.546371 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.548606 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.549966 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.550108 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.550212 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.550725 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.550855 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:31:55.551234 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.543715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.97.3:39329)
0620 02:31:55.544512 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.544533 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:55.544650 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:55.545056 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:55.545066 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:55.545086 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:31:55.545350 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.545357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.546530 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.546538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.548466 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:55.548478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.550349 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.550355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:55.550528 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.550568 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:31:55.550700 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 02:31:55.550830 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":20,"thread_start_us":470,"threads_started":1}
I20260620 02:31:55.551087 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4e8259064f4f2791ad3a1fd928b056" candidate_uuid: "0547e0c9d6644d158c773b75a26bdcff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f10482a2e041e28791219089d2f0fb" is_pre_election: true
I20260620 02:31:55.551651 15118 raft_consensus.cc:2454] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0547e0c9d6644d158c773b75a26bdcff in term 0.
I20260620 02:31:55.552452 15185 leader_election.cc:302] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0547e0c9d6644d158c773b75a26bdcff, 87f10482a2e041e28791219089d2f0fb; no voters:
I20260620 02:31:55.552698 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4e8259064f4f2791ad3a1fd928b056" candidate_uuid: "0547e0c9d6644d158c773b75a26bdcff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a561d764cb499b9980263ac9ebbede" is_pre_election: true
I20260620 02:31:55.552923 15316 raft_consensus.cc:2781] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:31:55.553129 15316 raft_consensus.cc:482] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:31:55.553251 15316 raft_consensus.cc:3037] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:55.553329 14986 raft_consensus.cc:2454] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0547e0c9d6644d158c773b75a26bdcff in term 0.
I20260620 02:31:55.555680 15316 raft_consensus.cc:504] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.556236 15316 leader_election.cc:288] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [CANDIDATE]: Term 1 election: Requested vote from peers 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215), 92a561d764cb499b9980263ac9ebbede (127.0.97.1:44569)
I20260620 02:31:55.556814 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4e8259064f4f2791ad3a1fd928b056" candidate_uuid: "0547e0c9d6644d158c773b75a26bdcff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f10482a2e041e28791219089d2f0fb"
I20260620 02:31:55.556847 14986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4e8259064f4f2791ad3a1fd928b056" candidate_uuid: "0547e0c9d6644d158c773b75a26bdcff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a561d764cb499b9980263ac9ebbede"
I20260620 02:31:55.557065 15118 raft_consensus.cc:3037] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:55.557085 14986 raft_consensus.cc:3037] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:31:55.559350 14986 raft_consensus.cc:2454] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0547e0c9d6644d158c773b75a26bdcff in term 1.
I20260620 02:31:55.559360 15118 raft_consensus.cc:2454] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0547e0c9d6644d158c773b75a26bdcff in term 1.
I20260620 02:31:55.559871 15185 leader_election.cc:302] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0547e0c9d6644d158c773b75a26bdcff, 92a561d764cb499b9980263ac9ebbede; no voters:
I20260620 02:31:55.560189 15316 raft_consensus.cc:2781] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:31:55.560679 15316 raft_consensus.cc:686] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 1 LEADER]: Becoming Leader. State: Replica: 0547e0c9d6644d158c773b75a26bdcff, State: Running, Role: LEADER
I20260620 02:31:55.561278 15316 consensus_queue.cc:237] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:55.565984 14846 catalog_manager.cc:5495] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff reported cstate change: term changed from 0 to 1, leader changed from <none> to 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0547e0c9d6644d158c773b75a26bdcff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } health_report { overall_health: UNKNOWN } } }
I20260620 02:31:55.595532 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.590213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.0.97.3:39031)
0620 02:31:55.590408 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.590412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.590429 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:31:55.590538 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.590542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.590618 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.590753 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:31:55.591248 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.591982 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.593487 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.593882 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.593983 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.594448 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:31:55.595043 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:31:55.595153 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.595252 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.595319 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:31:55.603974 14986 raft_consensus.cc:1261] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Refusing update from remote peer 0547e0c9d6644d158c773b75a26bdcff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:55.603991 15118 raft_consensus.cc:1261] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Refusing update from remote peer 0547e0c9d6644d158c773b75a26bdcff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:31:55.605095 15320 consensus_queue.cc:1035] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:31:55.605482 15316 consensus_queue.cc:1035] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:31:55.613054 15295 tablet.cc:2369] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:31:55.618814 15323 mvcc.cc:205] Tried to move back new op lower bound from 7298755443103973376 to 7298755442943307776. Current Snapshot: MvccSnapshot[applied={T|T < 7298755443103973376}]
I20260620 02:31:55.632166 14846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:31:55.632441 14846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:31:55.636368 14846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Sending DeleteTablet for 3 replicas of tablet 6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:31:55.637706 15229 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4e8259064f4f2791ad3a1fd928b056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47678
I20260620 02:31:55.638025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14883
I20260620 02:31:55.638141 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4e8259064f4f2791ad3a1fd928b056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33334
I20260620 02:31:55.638130 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4e8259064f4f2791ad3a1fd928b056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:31:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35146
I20260620 02:31:55.639691 15332 tablet_replica.cc:330] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: stopping tablet replica
I20260620 02:31:55.640179 15332 raft_consensus.cc:2229] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:31:55.641042 15332 raft_consensus.cc:2258] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:55.648753 15332 ts_tablet_manager.cc:1905] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:55.651774 15333 tablet_replica.cc:330] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: stopping tablet replica
I20260620 02:31:55.652539 15333 raft_consensus.cc:2229] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:31:55.653333 15333 raft_consensus.cc:2258] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:55.656200 15333 ts_tablet_manager.cc:1905] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:55.656424 15332 ts_tablet_manager.cc:1918] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:55.656625 15332 log.cc:1196] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:31:55.659428 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.639472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.0.97.1:44569)
0620 02:31:55.639661 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.639669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.639690 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:31:55.639752 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.639760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.639838 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.639953 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:55.642107 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.643351 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.645408 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.646238 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.646372 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.650503 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:55.650527 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:55.650546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:55.650597 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:55.654827 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.655818 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.655828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.655832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.655993 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.656507 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.656515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.656519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.657135 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:55.657495 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.658760 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.659017 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:31:55.660256 15026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:55.665688 15332 ts_tablet_manager.cc:1939] T 6d4e8259064f4f2791ad3a1fd928b056 P 0547e0c9d6644d158c773b75a26bdcff: Deleting consensus metadata
I20260620 02:31:55.667656 15333 ts_tablet_manager.cc:1918] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:31:55.667989 15333 log.cc:1196] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:31:55.668294 14830 catalog_manager.cc:4841] TS 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031): tablet 6d4e8259064f4f2791ad3a1fd928b056 (table test-table [id=4d19b5d0cd784a889157f22f8402292d]) successfully deleted
I20260620 02:31:55.668840 15333 ts_tablet_manager.cc:1939] T 6d4e8259064f4f2791ad3a1fd928b056 P 87f10482a2e041e28791219089d2f0fb: Deleting consensus metadata
I20260620 02:31:55.671277 14833 catalog_manager.cc:4841] TS 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215): tablet 6d4e8259064f4f2791ad3a1fd928b056 (table test-table [id=4d19b5d0cd784a889157f22f8402292d]) successfully deleted
I20260620 02:31:55.761647 15334 tablet_replica.cc:330] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: stopping tablet replica
I20260620 02:31:55.762204 15334 raft_consensus.cc:2229] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:31:55.762881 15334 raft_consensus.cc:2258] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:31:55.764911 15334 ts_tablet_manager.cc:1905] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:31:55.765380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14883
W20260620 02:31:55.782125 14833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44569 (error 108)
I20260620 02:31:55.782261 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 15035
I20260620 02:31:55.784439 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.783846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35158)
0620 02:31:55.784079 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.784238 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 02:31:55.784706 14833 catalog_manager.cc:4584] TS 92a561d764cb499b9980263ac9ebbede (127.0.97.1:44569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d4e8259064f4f2791ad3a1fd928b056: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
I20260620 02:31:55.799990 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.783189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.0.97.2:41215)
0620 02:31:55.783419 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.783425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.783727 (+ 302us) server_negotiation.cc:407] Connection header received
0620 02:31:55.783951 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.783958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.784056 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.784185 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:55.786699 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.787504 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.789598 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.790103 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.790226 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.793762 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:55.793778 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:55.793793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:55.793829 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:55.796908 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.797841 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.797848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.797850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.797992 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.798444 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.798448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.798450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.798837 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:55.799105 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.799477 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.799671 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 02:31:55.800843 15158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:55.838616 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.838030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35160)
0620 02:31:55.838291 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.838421 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 02:31:55.839802 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.839389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35172)
0620 02:31:55.839555 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.839646 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:31:55.892342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 15035
I20260620 02:31:55.900734 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.900083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35184)
0620 02:31:55.900450 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.900545 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 02:31:55.901927 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.901523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35196)
0620 02:31:55.901698 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.901799 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 02:31:55.916575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 15166
I20260620 02:31:55.932806 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.917377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.97.3:39031)
0620 02:31:55.917599 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:55.917605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:55.917686 (+ 81us) server_negotiation.cc:407] Connection header received
0620 02:31:55.917890 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:55.917896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:55.917973 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:55.918083 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:55.919591 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.920937 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.923029 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:55.923525 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:55.923669 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:55.926514 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:55.926527 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:55.926542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:55.926585 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:55.929804 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.930518 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.930532 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.930536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.930707 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:55.931112 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:55.931119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:55.931122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:55.931499 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:55.931833 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:55.932154 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:31:55.932381 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:31:55.933631 15289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:55.982911 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.982344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35198)
0620 02:31:55.982609 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.982737 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 02:31:55.984262 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:55.983738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35200)
0620 02:31:55.984067 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:55.984143 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260620 02:31:56.059080 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 15166
I20260620 02:31:56.079953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 14813
I20260620 02:31:56.081211 14876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:31:56.197561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 14813
I20260620 02:31:56.197641 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:56.196845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44569 (local address 127.0.0.1:35214)
0620 02:31:56.197215 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:56.197343 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260620 02:31:56.221854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44705
--webserver_interface=127.0.97.62
--webserver_port=35089
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:56.572929 15340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:56.573320 15340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:56.573469 15340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:56.581386 15340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:31:56.581508 15340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:56.581591 15340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:56.581681 15340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:31:56.581755 15340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:31:56.592494 15340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:56.594053 15340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:56.595445 15340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:56.608368 15345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:56.608423 15346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:56.608486 15348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:56.609480 15340 server_base.cc:1034] running on GCE node
I20260620 02:31:56.610431 15340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:56.612022 15340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:56.613253 15340 hybrid_clock.cc:648] HybridClock initialized: now 1781922716613212 us; error 42 us; skew 500 ppm
I20260620 02:31:56.613662 15340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:56 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:56.616907 15340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:31:56.618556 15340 webserver.cc:460] Webserver started at http://127.0.97.62:35089/ using document root <none> and password file <none>
I20260620 02:31:56.619234 15340 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:56.619343 15340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:56.625083 15340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:31:56.628021 15355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:56.629464 15340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:31:56.629652 15340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "1485a16502cb49949e9518de3112c0a2"
format_stamp: "Formatted at 2026-06-20 02:31:50 on dist-test-slave-r706"
server_key: "7141c377b3b2ca43de3410aace0a1999"
server_key_iv: "86519d818ad34657fe9e7aaaeb5009e2"
server_key_version: "encryptionkey@0"
I20260620 02:31:56.630666 15340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:56.660154 15340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:56.664422 15340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:56.664747 15340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:56.684167 15340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44705
I20260620 02:31:56.684201 15406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44705 every 8 connection(s)
I20260620 02:31:56.685748 15340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:31:56.689446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15340
I20260620 02:31:56.690274 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44569
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44705
--webserver_port=46265
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:31:56.692963 15407 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:31:56.696450 15407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap starting.
I20260620 02:31:56.717561 15407 log.cc:824] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:56.734066 15407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:31:56.734679 15407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2: Bootstrap complete.
I20260620 02:31:56.744180 15407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:56.745172 15407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1485a16502cb49949e9518de3112c0a2, State: Initialized, Role: FOLLOWER
I20260620 02:31:56.745807 15407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:56.746045 15407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:31:56.746162 15407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:31:56.746364 15407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:31:56.750272 15407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:56.750790 15407 leader_election.cc:302] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1485a16502cb49949e9518de3112c0a2; no voters:
I20260620 02:31:56.751485 15407 leader_election.cc:288] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:31:56.751634 15411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:31:56.752885 15411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [term 3 LEADER]: Becoming Leader. State: Replica: 1485a16502cb49949e9518de3112c0a2, State: Running, Role: LEADER
I20260620 02:31:56.753458 15411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } }
I20260620 02:31:56.753818 15407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:31:56.755977 15413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1485a16502cb49949e9518de3112c0a2. Latest consensus state: current_term: 3 leader_uuid: "1485a16502cb49949e9518de3112c0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } } }
I20260620 02:31:56.756340 15413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:56.756879 15412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1485a16502cb49949e9518de3112c0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1485a16502cb49949e9518de3112c0a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44705 } } }
I20260620 02:31:56.757185 15412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1485a16502cb49949e9518de3112c0a2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:31:56.762490 15417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:31:56.773581 15417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:31:56.776003 15417 catalog_manager.cc:1259] Loaded cluster ID: 84c7ba71dd894e85a4e00e7d6b57ff49
I20260620 02:31:56.776149 15417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 02:31:56.779099 15417 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa75c4979ad google::LogMessage::Fail() at ??:0
@ 0x7fa75c49a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa75c4974de google::LogMessage::Flush() at ??:0
@ 0x7fa75c498a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa771ece512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa771ecb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa771f53610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa771f53439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa75f0e3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa75f0e691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa75f0e6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa75f0c1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa76239b6db start_thread at ??:0
@ 0x7fa75b33161f clone at ??:0
W20260620 02:31:57.122402 15409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:57.122711 15409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:57.122807 15409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:57.130273 15409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:57.130390 15409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:57.130507 15409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:31:57.140625 15409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46265
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:57.141987 15409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:57.143347 15409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:57.157064 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:57.157025 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:57.156967 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:57.157992 15409 server_base.cc:1034] running on GCE node
I20260620 02:31:57.158787 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:57.159807 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:57.161106 15409 hybrid_clock.cc:648] HybridClock initialized: now 1781922717161053 us; error 41 us; skew 500 ppm
I20260620 02:31:57.161464 15409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:57 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:57.164865 15409 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:31:57.166563 15409 webserver.cc:460] Webserver started at http://127.0.97.1:46265/ using document root <none> and password file <none>
I20260620 02:31:57.167302 15409 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:57.167460 15409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:57.173400 15409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:31:57.176672 15458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:57.178238 15409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:31:57.178445 15409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "92a561d764cb499b9980263ac9ebbede"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "413904337c9e2c005d074e487451a7e6"
server_key_iv: "4df11c87e9a7596be7d3a40f6051b325"
server_key_version: "encryptionkey@0"
I20260620 02:31:57.179514 15409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:57.192538 15409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:57.196986 15409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:57.197402 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:57.198442 15409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:57.200444 15465 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 02:31:57.207274 15409 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 02:31:57.207451 15409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 02:31:57.207598 15409 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 02:31:57.211009 15409 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 02:31:57.211123 15409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260620 02:31:57.211505 15465 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Bootstrap starting.
I20260620 02:31:57.220434 15465 log.cc:824] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Log is configured to *not* fsync() on all Append() calls
I20260620 02:31:57.236719 15465 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:31:57.237479 15465 tablet_bootstrap.cc:492] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Bootstrap complete.
I20260620 02:31:57.237973 15465 ts_tablet_manager.cc:1397] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.011s
I20260620 02:31:57.247488 15465 raft_consensus.cc:348] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:57.249271 15465 raft_consensus.cc:729] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a561d764cb499b9980263ac9ebbede, State: Initialized, Role: FOLLOWER
I20260620 02:31:57.250020 15465 consensus_queue.cc:260] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:57.251688 15465 ts_tablet_manager.cc:1428] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260620 02:31:57.259013 15409 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44569
I20260620 02:31:57.259093 15573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44569 every 8 connection(s)
I20260620 02:31:57.260362 15409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:31:57.264870 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:57.263080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:44785)
0620 02:31:57.263838 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:57.264027 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":316,"threads_started":1}
W20260620 02:31:57.265558 15574 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44705 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44705: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
I20260620 02:31:57.266369 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:57.266056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:41241)
0620 02:31:57.266199 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:57.266266 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 02:31:57.267177 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:57.266868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:47673)
0620 02:31:57.266992 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:57.267058 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 02:31:57.267414 15574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:31:57.268088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15409
I20260620 02:31:57.268713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41215
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44705
--webserver_port=33467
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:57.600967 15577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:57.601389 15577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:57.601595 15577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:57.609052 15577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:57.609218 15577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:57.609424 15577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:31:57.619457 15577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33467
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:57.620857 15577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:57.622246 15577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:57.635860 15583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:57.635967 15585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:57.635942 15582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:57.637071 15577 server_base.cc:1034] running on GCE node
I20260620 02:31:57.637792 15577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:57.638870 15577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:57.640107 15577 hybrid_clock.cc:648] HybridClock initialized: now 1781922717640053 us; error 41 us; skew 500 ppm
I20260620 02:31:57.640468 15577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:57 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:57.643905 15577 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:31:57.645534 15577 webserver.cc:460] Webserver started at http://127.0.97.2:33467/ using document root <none> and password file <none>
I20260620 02:31:57.646242 15577 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:57.646400 15577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:57.651824 15577 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:31:57.654585 15592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:57.655946 15577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:31:57.656128 15577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "87f10482a2e041e28791219089d2f0fb"
format_stamp: "Formatted at 2026-06-20 02:31:51 on dist-test-slave-r706"
server_key: "17d3724078e28482b9b1969fe859943d"
server_key_iv: "0caa9a333875c0796bab8d0b4eb95e48"
server_key_version: "encryptionkey@0"
I20260620 02:31:57.657176 15577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:57.674934 15577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:57.679279 15577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:57.679581 15577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:57.680637 15577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:57.682103 15577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:57.682240 15577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:57.682369 15577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:57.682463 15577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:57.717620 15577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41215
I20260620 02:31:57.717717 15704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41215 every 8 connection(s)
I20260620 02:31:57.719260 15577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:31:57.724488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15577
I20260620 02:31:57.724509 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:57.722703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:44729)
0620 02:31:57.723435 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:57.723629 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":294,"threads_started":1}
I20260620 02:31:57.725332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39031
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44705
--webserver_port=38113
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:31:57.725384 15705 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44705 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44705: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
I20260620 02:31:57.726311 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:57.725968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:35753)
0620 02:31:57.726129 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:57.726205 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 02:31:57.727249 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:57.726914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:56491)
0620 02:31:57.727055 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:57.727136 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:31:57.727483 15705 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:31:58.065317 15708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:31:58.065613 15708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:31:58.065802 15708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:31:58.073118 15708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:31:58.073249 15708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:31:58.073442 15708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:31:58.083803 15708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38113
--tserver_master_addrs=127.0.97.62:44705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:31:58.085204 15708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:31:58.086566 15708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:31:58.100163 15714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:58.100240 15716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:31:58.100210 15713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:31:58.101153 15708 server_base.cc:1034] running on GCE node
I20260620 02:31:58.101784 15708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:31:58.102753 15708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:31:58.103992 15708 hybrid_clock.cc:648] HybridClock initialized: now 1781922718103933 us; error 56 us; skew 500 ppm
I20260620 02:31:58.104318 15708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:31:58 dist-test-slave-r706 krb5kdc[14258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:31:58.107599 15708 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:31:58.109160 15708 webserver.cc:460] Webserver started at http://127.0.97.3:38113/ using document root <none> and password file <none>
I20260620 02:31:58.109833 15708 fs_manager.cc:359] Metadata directory not provided
I20260620 02:31:58.109935 15708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:31:58.115250 15708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:31:58.117926 15723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:58.119395 15708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:31:58.119612 15708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "0547e0c9d6644d158c773b75a26bdcff"
format_stamp: "Formatted at 2026-06-20 02:31:52 on dist-test-slave-r706"
server_key: "78e28ac9eebf79062fc6780a3400feb6"
server_key_iv: "94f94d588fdd3747afcc2f0222542737"
server_key_version: "encryptionkey@0"
I20260620 02:31:58.120815 15708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:31:58.153076 15708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:31:58.157644 15708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:31:58.157943 15708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:31:58.159049 15708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:31:58.160630 15708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:31:58.160828 15708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:58.160934 15708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:31:58.161015 15708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:31:58.195396 15708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39031
I20260620 02:31:58.195485 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39031 every 8 connection(s)
I20260620 02:31:58.196943 15708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:31:58.202059 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.200127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:37851)
0620 02:31:58.200863 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.201176 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":311,"threads_started":1}
I20260620 02:31:58.202759 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15708
W20260620 02:31:58.202912 15836 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44705 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44705: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
I20260620 02:31:58.204007 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.203661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:55391)
0620 02:31:58.203825 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.203897 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:31:58.205111 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.204659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:49165)
0620 02:31:58.204829 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.204924 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 02:31:58.205374 15836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:31:58.268654 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.267945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:56811)
0620 02:31:58.268377 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.268462 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":210,"threads_started":1}
I20260620 02:31:58.728919 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.728097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:47421)
0620 02:31:58.728593 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.728686 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260620 02:31:58.780301 15842 raft_consensus.cc:482] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:31:58.780673 15842 raft_consensus.cc:504] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:31:58.782322 15842 leader_election.cc:288] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215), 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031)
Jun 20 02:31:58 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 02:31:58 dist-test-slave-r706 krb5kdc[14258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:31:58.809124 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.782997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41823 (local address 127.0.97.3:39031)
0620 02:31:58.783612 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:58.783636 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:58.783700 (+ 64us) server_negotiation.cc:407] Connection header received
0620 02:31:58.786436 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:58.786445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:58.786804 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:58.787159 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:58.789213 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.791209 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.794207 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.795263 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.795470 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:58.801737 (+ 6267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:58.801746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:58.801762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:58.801804 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:58.804967 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:58.805964 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:58.805974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:58.805983 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:58.806164 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:58.806636 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:58.806641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:58.806645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:58.807206 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:58.807423 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:58.808277 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 02:31:58.808655 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":414,"threads_started":1}
I20260620 02:31:58.809135 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.783056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45665 (local address 127.0.97.2:41215)
0620 02:31:58.784534 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0620 02:31:58.784549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:31:58.784607 (+ 58us) server_negotiation.cc:407] Connection header received
0620 02:31:58.786377 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:31:58.786385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:31:58.786769 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:31:58.787081 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:31:58.789279 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.791209 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.794207 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.795264 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.795494 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:58.801765 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:31:58.801773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:31:58.801786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:31:58.801831 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:31:58.804970 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:58.805975 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:58.805981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:58.805987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:58.806159 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:31:58.806665 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:31:58.806668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:31:58.806670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:31:58.807211 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:31:58.807422 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:31:58.808247 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 02:31:58.808663 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":1036,"threads_started":1}
I20260620 02:31:58.809154 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.782648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39031 (local address 127.0.97.1:41823)
0620 02:31:58.783209 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.783256 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:58.784458 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:58.787280 (+ 2822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:58.787291 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:58.788171 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:58.788960 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:58.788980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.791477 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.791481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:58.794020 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:58.794039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.795568 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.795577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:58.795978 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:58.797064 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:58.797118 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:58.801362 (+ 4244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:58.805208 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:58.805218 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:58.805236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:58.805800 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:58.806324 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:58.806327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:58.806330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:58.806507 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:58.807356 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:58.807365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:58.807728 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 02:31:58.808020 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 02:31:58.808645 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":344,"spinlock_wait_cycles":54272,"thread_start_us":256,"threads_started":1}
I20260620 02:31:58.809526 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:58.782655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41215 (local address 127.0.97.1:45665)
0620 02:31:58.784286 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:58.784305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:31:58.784486 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:31:58.787283 (+ 2797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:31:58.787293 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:31:58.788177 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:31:58.788956 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:58.788978 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.791472 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.791479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:58.794020 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:31:58.794039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:31:58.795579 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:31:58.795586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:31:58.795977 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:31:58.797064 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:31:58.797118 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:31:58.801494 (+ 4376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:31:58.805208 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:58.805218 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:58.805236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:58.805804 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:58.806288 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:31:58.806297 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:31:58.806301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:31:58.806515 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:31:58.807357 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:31:58.807366 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:31:58.807727 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 02:31:58.808021 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 02:31:58.808849 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"mutex_wait_us":1244,"spinlock_wait_cycles":103552,"thread_start_us":712,"threads_started":1}
W20260620 02:31:58.812794 15462 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
W20260620 02:31:58.813553 15459 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:31:58.813697 15459 leader_election.cc:302] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a561d764cb499b9980263ac9ebbede; no voters: 0547e0c9d6644d158c773b75a26bdcff, 87f10482a2e041e28791219089d2f0fb
I20260620 02:31:58.814172 15842 raft_consensus.cc:2726] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:31:59.206965 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:59.205949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:42941)
0620 02:31:59.206477 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:59.206645 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":215,"threads_started":1}
I20260620 02:31:59.270193 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:59.269726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:38899)
0620 02:31:59.269933 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:59.270048 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 02:31:59.730614 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:31:59.729898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:48301)
0620 02:31:59.730371 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 02:31:59.730453 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
I20260620 02:32:00.209475 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:00.208519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:52885)
0620 02:32:00.209199 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:00.209304 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":335,"threads_started":1}
I20260620 02:32:00.272013 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:00.271235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:41377)
0620 02:32:00.271725 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:00.271811 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
I20260620 02:32:00.448284 15851 raft_consensus.cc:482] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:32:00.448518 15851 raft_consensus.cc:504] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:32:00.449465 15851 leader_election.cc:288] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215), 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031)
W20260620 02:32:00.450543 15462 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
W20260620 02:32:00.450799 15459 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:32:00.450946 15459 leader_election.cc:302] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a561d764cb499b9980263ac9ebbede; no voters: 0547e0c9d6644d158c773b75a26bdcff, 87f10482a2e041e28791219089d2f0fb
I20260620 02:32:00.451221 15851 raft_consensus.cc:2726] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:32:00.732483 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:00.731661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:52319)
0620 02:32:00.732170 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:00.732267 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":281,"threads_started":1}
I20260620 02:32:01.211345 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:01.210597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:49255)
0620 02:32:01.211092 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:01.211176 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
I20260620 02:32:01.273895 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:01.273137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:40687)
0620 02:32:01.273608 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:01.273705 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260620 02:32:01.734465 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:01.733590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:38549)
0620 02:32:01.734168 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:01.734272 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260620 02:32:02.213503 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:02.212486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:57179)
0620 02:32:02.213150 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:02.213290 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":347,"threads_started":1}
I20260620 02:32:02.275955 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:02.275133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:36365)
0620 02:32:02.275660 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:02.275745 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":245,"threads_started":1}
I20260620 02:32:02.565927 15858 raft_consensus.cc:482] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:32:02.566207 15858 raft_consensus.cc:504] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:32:02.566896 15858 leader_election.cc:288] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215), 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031)
W20260620 02:32:02.568049 15462 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
W20260620 02:32:02.568289 15459 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:32:02.568390 15459 leader_election.cc:302] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a561d764cb499b9980263ac9ebbede; no voters: 0547e0c9d6644d158c773b75a26bdcff, 87f10482a2e041e28791219089d2f0fb
I20260620 02:32:02.568624 15858 raft_consensus.cc:2726] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:32:02.736403 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:02.735627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:34275)
0620 02:32:02.736140 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:02.736237 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":288,"threads_started":1}
I20260620 02:32:03.215389 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:03.214622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:50667)
0620 02:32:03.215128 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:03.215216 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260620 02:32:03.277937 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:03.277125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:55827)
0620 02:32:03.277679 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:03.277770 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":263,"threads_started":1}
I20260620 02:32:03.739749 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:03.737983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:34245)
0620 02:32:03.739166 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:03.739317 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":517,"threads_started":1}
I20260620 02:32:04.217626 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:04.216600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:41947)
0620 02:32:04.217359 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:04.217444 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":469,"threads_started":1}
I20260620 02:32:04.280570 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:04.279291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:42225)
0620 02:32:04.280228 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:04.280349 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":465,"threads_started":1}
I20260620 02:32:04.742333 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:04.741308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:43575)
0620 02:32:04.741896 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:04.742062 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":281,"threads_started":1}
I20260620 02:32:05.219874 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:05.218949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:38339)
0620 02:32:05.219574 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:05.219671 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":296,"threads_started":1}
I20260620 02:32:05.282716 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:05.281904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:53123)
0620 02:32:05.282446 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:05.282543 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":285,"threads_started":1}
I20260620 02:32:05.744164 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:05.743423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:36247)
0620 02:32:05.743898 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:05.743986 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260620 02:32:06.221875 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:06.221099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:49989)
0620 02:32:06.221576 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:06.221666 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
I20260620 02:32:06.284601 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:06.283705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:40297)
0620 02:32:06.284315 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:06.284408 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":280,"threads_started":1}
I20260620 02:32:06.745993 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:06.745227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:50875)
0620 02:32:06.745716 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:06.745825 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260620 02:32:07.223726 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:07.222963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:60141)
0620 02:32:07.223478 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:07.223568 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":247,"threads_started":1}
I20260620 02:32:07.286442 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:07.285719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:40575)
0620 02:32:07.286191 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:07.286275 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":235,"threads_started":1}
I20260620 02:32:07.517426 15874 raft_consensus.cc:482] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:32:07.517654 15874 raft_consensus.cc:504] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f10482a2e041e28791219089d2f0fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41215 } } peers { permanent_uuid: "0547e0c9d6644d158c773b75a26bdcff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39031 } } peers { permanent_uuid: "92a561d764cb499b9980263ac9ebbede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44569 } }
I20260620 02:32:07.518379 15874 leader_election.cc:288] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215), 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031)
W20260620 02:32:07.519467 15462 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f10482a2e041e28791219089d2f0fb (127.0.97.2:41215): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
W20260620 02:32:07.519722 15459 leader_election.cc:341] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0547e0c9d6644d158c773b75a26bdcff (127.0.97.3:39031): Not found: Tablet not found: 6d4e8259064f4f2791ad3a1fd928b056
I20260620 02:32:07.519870 15459 leader_election.cc:302] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a561d764cb499b9980263ac9ebbede; no voters: 0547e0c9d6644d158c773b75a26bdcff, 87f10482a2e041e28791219089d2f0fb
I20260620 02:32:07.520149 15874 raft_consensus.cc:2726] T 6d4e8259064f4f2791ad3a1fd928b056 P 92a561d764cb499b9980263ac9ebbede [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:32:07.748061 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:07.747193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:41719)
0620 02:32:07.747749 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:07.747834 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":303,"threads_started":1}
I20260620 02:32:08.225986 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:08.225142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:40179)
0620 02:32:08.225680 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:08.225777 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
I20260620 02:32:08.288264 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:08.287488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:41137)
0620 02:32:08.287986 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:08.288076 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
I20260620 02:32:08.750028 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:08.749264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:33477)
0620 02:32:08.749766 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:08.749853 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":230,"threads_started":1}
I20260620 02:32:09.227691 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:09.226933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:58719)
0620 02:32:09.227433 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:09.227524 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260620 02:32:09.290220 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:09.289342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:37301)
0620 02:32:09.289909 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:09.290027 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":345,"threads_started":1}
I20260620 02:32:09.751997 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:09.751181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:45633)
0620 02:32:09.751715 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:09.751802 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":266,"threads_started":1}
I20260620 02:32:10.229590 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:10.228657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:42517)
0620 02:32:10.229234 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:10.229353 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
I20260620 02:32:10.292044 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:10.291271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:42479)
0620 02:32:10.291766 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:10.291864 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
I20260620 02:32:10.753890 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:10.753094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:45259)
0620 02:32:10.753623 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:10.753716 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
I20260620 02:32:11.231397 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:11.230640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:43051)
0620 02:32:11.231125 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:11.231216 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":235,"threads_started":1}
I20260620 02:32:11.294020 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:11.293208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:41865)
0620 02:32:11.293716 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:11.293815 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":272,"threads_started":1}
I20260620 02:32:11.755797 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:11.754984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:33395)
0620 02:32:11.755501 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:11.755589 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260620 02:32:12.233816 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:12.232763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:37567)
0620 02:32:12.233523 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:12.233612 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":577,"threads_started":1}
I20260620 02:32:12.296162 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:12.295382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:33967)
0620 02:32:12.295867 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:12.295959 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260620 02:32:12.757954 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:12.757182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.2:60323)
0620 02:32:12.757684 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:12.757782 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260620 02:32:13.210238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 15409
I20260620 02:32:13.226161 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:13.211591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.0.97.1:44569)
0620 02:32:13.212193 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:13.212201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:13.212227 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:32:13.212287 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:13.212294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:13.212513 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:13.212690 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:13.214304 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:13.215280 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:13.216578 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:13.217231 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:13.217370 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:13.220210 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:13.220222 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:13.220237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:13.220275 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:13.223207 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:13.223923 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:13.223930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:13.223932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:13.224074 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:13.224470 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:13.224478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:13.224481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:13.224871 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:13.225209 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:13.225549 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 02:32:13.225828 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":313,"threads_started":1}
I20260620 02:32:13.227423 15569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:13.235832 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:13.235118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.3:60727)
0620 02:32:13.235575 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:13.235662 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260620 02:32:13.320266 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:13.318410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44705 (local address 127.0.97.1:56375)
0620 02:32:13.319724 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:13.319874 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":779,"threads_started":1}
I20260620 02:32:13.321376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 15409
I20260620 02:32:13.342511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 15577
I20260620 02:32:13.356458 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:13.343380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.0.97.2:41215)
0620 02:32:13.343912 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:13.343916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:13.343933 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:32:13.343989 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:13.343993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:13.344051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:13.344145 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:13.345781 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:13.346628 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:13.347948 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:13.348389 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:13.348494 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:13.351194 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:13.351200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:13.351203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:13.351236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:13.353948 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:13.354670 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:13.354674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:13.354677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:13.354813 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:13.355156 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:13.355160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:13.355162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:13.355454 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:13.355662 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:13.355942 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 02:32:13.356128 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":295,"threads_started":1}
I20260620 02:32:13.357380 15700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:13.447928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 15577
I20260620 02:32:13.466586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 15708
I20260620 02:32:13.480719 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:13.467518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.97.3:39031)
0620 02:32:13.468097 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:13.468102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:13.468120 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:32:13.468177 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:13.468180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:13.468243 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:13.468369 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:13.469956 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:13.470754 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:13.471945 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:13.472398 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:13.472494 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:13.475072 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:13.475077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:13.475080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:13.475110 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:13.477985 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:13.478766 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:13.478773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:13.478775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:13.478907 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:13.479262 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:13.479266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:13.479268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:13.479564 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:13.479796 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:13.480099 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 02:32:13.480277 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":302,"threads_started":1}
I20260620 02:32:13.481446 15831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:13.565315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 15708
2026-06-20T02:32:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24946 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:32:13 dist-test-slave-r706 krb5kdc[15901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:32:13 dist-test-slave-r706 krb5kdc[15901](info): set up 2 sockets
Jun 20 02:32:13 dist-test-slave-r706 krb5kdc[15901](info): commencing operation
krb5kdc: starting...
W20260620 02:32:14.689779 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:32:14 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:32:14.961855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45875
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:32:15.306038 15914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:15.306391 15914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:15.306560 15914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:15.314378 15914 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:32:15.314575 15914 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:32:15.314700 15914 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:32:15.314797 15914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:32:15.314908 15914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:15.315037 15914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:15.315128 15914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:32:15.315218 15914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:32:15.328271 15914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:15.330083 15914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:15.331653 15914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:15.344540 15919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:15.344626 15922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:15.344579 15920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:15.345329 15914 server_base.cc:1034] running on GCE node
I20260620 02:32:15.346210 15914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:15.347707 15914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:15.349097 15914 hybrid_clock.cc:648] HybridClock initialized: now 1781922735348961 us; error 43 us; skew 500 ppm
I20260620 02:32:15.349596 15914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:15 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:15.353327 15914 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:32:15.354945 15914 webserver.cc:460] Webserver started at http://127.0.97.62:35685/ using document root <none> and password file <none>
I20260620 02:32:15.355778 15914 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:15.356067 15914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:15.356462 15914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:32:15.359884 15914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "61f8392a77a7401b905389746eb42a7e"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "90137b48f52bef734064b580c894ffc8"
server_key_iv: "3b7a9bd35320ceb6f1c7153deef463bd"
server_key_version: "encryptionkey@0"
I20260620 02:32:15.360764 15914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "61f8392a77a7401b905389746eb42a7e"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "90137b48f52bef734064b580c894ffc8"
server_key_iv: "3b7a9bd35320ceb6f1c7153deef463bd"
server_key_version: "encryptionkey@0"
I20260620 02:32:15.366508 15914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:32:15.370500 15929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:15.372339 15914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:32:15.372516 15914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "61f8392a77a7401b905389746eb42a7e"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "90137b48f52bef734064b580c894ffc8"
server_key_iv: "3b7a9bd35320ceb6f1c7153deef463bd"
server_key_version: "encryptionkey@0"
I20260620 02:32:15.372730 15914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:15.390125 15914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:15.394606 15914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:15.394853 15914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:15.414350 15914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45875
I20260620 02:32:15.414374 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45875 every 8 connection(s)
I20260620 02:32:15.415809 15914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:32:15.419188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 15914
I20260620 02:32:15.419411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:32:15.420033 15981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:15.420233 388 external_mini_cluster.cc:1428] Setting key ba395162df01c5596a4e9faae2bed5e2
I20260620 02:32:15.433816 15981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap starting.
Jun 20 02:32:15 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:15.437440 15981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:15.438645 15981 log.cc:824] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:15.441854 15981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: No bootstrap required, opened a new log
I20260620 02:32:15.443377 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:15.422468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.0.97.62:45875)
0620 02:32:15.423356 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:15.423368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:15.423411 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:32:15.425272 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:15.425294 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:15.425701 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:15.426120 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:15.427651 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:15.428934 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:15.430325 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:15.431021 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:15.431389 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:15.435890 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:15.435899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:15.435917 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:15.435957 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:15.439195 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:15.439956 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:15.439964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:15.439970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:15.440173 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:15.440572 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:15.440577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:15.440580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:15.441191 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:15.441478 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:15.441968 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 02:32:15.442402 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":375,"threads_started":1}
I20260620 02:32:15.451521 15981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:15.451967 15981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:15.452149 15981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8392a77a7401b905389746eb42a7e, State: Initialized, Role: FOLLOWER
I20260620 02:32:15.452785 15981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:15.453090 15981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:32:15.453271 15981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:32:15.453465 15981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:15.456255 15981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:15.456782 15981 leader_election.cc:302] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f8392a77a7401b905389746eb42a7e; no voters:
I20260620 02:32:15.457451 15981 leader_election.cc:288] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:32:15.457612 15986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:32:15.458408 15986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 LEADER]: Becoming Leader. State: Replica: 61f8392a77a7401b905389746eb42a7e, State: Running, Role: LEADER
I20260620 02:32:15.458942 15986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:15.459743 15981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:32:15.463045 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f8392a77a7401b905389746eb42a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } } }
I20260620 02:32:15.463824 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: This master's current role is: LEADER
I20260620 02:32:15.463332 15988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f8392a77a7401b905389746eb42a7e. Latest consensus state: current_term: 1 leader_uuid: "61f8392a77a7401b905389746eb42a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } } }
I20260620 02:32:15.465420 15988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: This master's current role is: LEADER
I20260620 02:32:15.470419 15993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:32:15.476171 15993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:32:15.485894 15993 catalog_manager.cc:1347] Generated new cluster ID: 4e69bc904c3340ab860474d6bf69f6d5
I20260620 02:32:15.486033 15993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:32:15.503634 15993 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:32:15.504987 15993 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:32:15.514060 15993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Generated new TSK 0
I20260620 02:32:15.514974 15993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:32:15.584237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45875
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:32:15.939991 16007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:15.940271 16007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:15.940434 16007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:15.947746 16007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:15.947852 16007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:15.948031 16007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:32:15.958155 16007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:15.959427 16007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:15.960817 16007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:15.974376 16012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:15.974344 16013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:15.974411 16015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:15.975379 16007 server_base.cc:1034] running on GCE node
I20260620 02:32:15.976008 16007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:15.977062 16007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:15.978302 16007 hybrid_clock.cc:648] HybridClock initialized: now 1781922735978238 us; error 57 us; skew 500 ppm
I20260620 02:32:15.978636 16007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:15 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:15.982172 16007 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:32:15.983673 16007 webserver.cc:460] Webserver started at http://127.0.97.1:43575/ using document root <none> and password file <none>
I20260620 02:32:15.984417 16007 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:15.984575 16007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:15.984930 16007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:32:15.988220 16007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "e59be24c5e994c2988f086fe7eb2ed86"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "f0b019b5b5789d8e7b1c18a7b1b3ffdc"
server_key_iv: "0c2b623a63069e5b708a0799e589ecbd"
server_key_version: "encryptionkey@0"
I20260620 02:32:15.989156 16007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e59be24c5e994c2988f086fe7eb2ed86"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "f0b019b5b5789d8e7b1c18a7b1b3ffdc"
server_key_iv: "0c2b623a63069e5b708a0799e589ecbd"
server_key_version: "encryptionkey@0"
I20260620 02:32:15.995774 16007 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:32:15.999611 16022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:16.001125 16007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:32:16.001319 16007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "e59be24c5e994c2988f086fe7eb2ed86"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "f0b019b5b5789d8e7b1c18a7b1b3ffdc"
server_key_iv: "0c2b623a63069e5b708a0799e589ecbd"
server_key_version: "encryptionkey@0"
I20260620 02:32:16.001540 16007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:16.017169 16007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:16.021632 16007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:16.021984 16007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:16.023108 16007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:32:16.024652 16007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:32:16.024739 16007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:16.024919 16007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:32:16.025044 16007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:16.061884 16007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34655
I20260620 02:32:16.061944 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34655 every 8 connection(s)
I20260620 02:32:16.063280 16007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:32:16.065325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16007
I20260620 02:32:16.065577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:32:16.066479 388 external_mini_cluster.cc:1428] Setting key da9a339f9f52b7a45136328d9b99d5f6
Jun 20 02:32:16 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:16.086359 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:16.066227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44855 (local address 127.0.97.62:45875)
0620 02:32:16.066730 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:16.066734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:16.067969 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 02:32:16.069967 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:16.069976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:16.070118 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:16.070238 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:16.072005 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.073229 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.074880 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.075608 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.075766 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:16.080215 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:16.080221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:16.080225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:16.080292 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:16.082986 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:16.083801 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:16.083807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:16.083810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:16.083946 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:16.084414 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:16.084420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:16.084424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:16.084712 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:16.084879 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:16.085719 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 02:32:16.085969 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
I20260620 02:32:16.087442 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:16.066852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:44855)
0620 02:32:16.067798 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:16.067850 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:16.069684 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:16.070489 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:16.070498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:16.071091 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:32:16.071774 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:16.071819 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.073409 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.073414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:16.074717 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:16.074729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.075864 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.075870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:16.076087 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:16.076903 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:32:16.076926 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:32:16.079882 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:32:16.083151 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:16.083161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:16.083181 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:16.083650 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:16.084073 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:16.084078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:16.084081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:16.084288 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:16.084856 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:32:16.084865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:32:16.085320 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 02:32:16.085615 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 02:32:16.086214 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":411,"threads_started":1}
I20260620 02:32:16.089305 16135 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45875
I20260620 02:32:16.089679 16135 heartbeater.cc:464] Registering TS with master...
I20260620 02:32:16.090716 16135 heartbeater.cc:511] Master 127.0.97.62:45875 requested a full tablet report, sending...
I20260620 02:32:16.093487 15946 ts_manager.cc:194] Registered new tserver with Master: e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1:34655)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:32:16.095201 15946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:32:16.128357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45875
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:32:16.457139 16141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:16.457430 16141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:16.457592 16141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:16.464951 16141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:16.465088 16141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:16.465274 16141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:32:16.475250 16141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:16.476608 16141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:16.478071 16141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:16.491299 16146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:16.491322 16147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:16.491285 16149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:16.492658 16141 server_base.cc:1034] running on GCE node
I20260620 02:32:16.493342 16141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:16.494333 16141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:16.495536 16141 hybrid_clock.cc:648] HybridClock initialized: now 1781922736495488 us; error 37 us; skew 500 ppm
I20260620 02:32:16.495900 16141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:16 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:16.499338 16141 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:32:16.500885 16141 webserver.cc:460] Webserver started at http://127.0.97.2:39289/ using document root <none> and password file <none>
I20260620 02:32:16.501643 16141 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:16.501754 16141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:16.502024 16141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:32:16.505486 16141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "8d7640c157bb43818ad68268ebee8719"
format_stamp: "Formatted at 2026-06-20 02:32:16 on dist-test-slave-r706"
server_key: "d1d7c822960c382aaf46499e74f4887d"
server_key_iv: "47ec786077985440b34c995b083bce48"
server_key_version: "encryptionkey@0"
I20260620 02:32:16.506386 16141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8d7640c157bb43818ad68268ebee8719"
format_stamp: "Formatted at 2026-06-20 02:32:16 on dist-test-slave-r706"
server_key: "d1d7c822960c382aaf46499e74f4887d"
server_key_iv: "47ec786077985440b34c995b083bce48"
server_key_version: "encryptionkey@0"
I20260620 02:32:16.511925 16141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:32:16.515626 16156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:16.517050 16141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:32:16.517261 16141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "8d7640c157bb43818ad68268ebee8719"
format_stamp: "Formatted at 2026-06-20 02:32:16 on dist-test-slave-r706"
server_key: "d1d7c822960c382aaf46499e74f4887d"
server_key_iv: "47ec786077985440b34c995b083bce48"
server_key_version: "encryptionkey@0"
I20260620 02:32:16.517452 16141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:16.541880 16141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:16.546232 16141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:16.546540 16141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:16.547552 16141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:32:16.548920 16141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:32:16.549038 16141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:16.549194 16141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:32:16.549253 16141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:16.583581 16141 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42077
I20260620 02:32:16.583674 16268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42077 every 8 connection(s)
I20260620 02:32:16.584914 16141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:32:16.594497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16141
I20260620 02:32:16.594698 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:32:16.595189 388 external_mini_cluster.cc:1428] Setting key fbfde208bc261200856c63b45edea257
Jun 20 02:32:16 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:16.605387 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:16.587609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53159 (local address 127.0.97.62:45875)
0620 02:32:16.588071 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:16.588075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:16.588797 (+ 722us) server_negotiation.cc:407] Connection header received
0620 02:32:16.590726 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:16.590732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:16.590827 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:16.590951 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:16.592437 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.593230 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.594814 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.595329 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.595461 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:16.599368 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:16.599374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:16.599377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:16.599411 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:16.602063 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:16.602850 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:16.602857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:16.602861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:16.602996 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:16.603456 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:16.603461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:16.603465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:16.603762 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:16.603926 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:16.604674 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 02:32:16.604914 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":259,"threads_started":1}
I20260620 02:32:16.606525 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:16.587947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:53159)
0620 02:32:16.588623 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:16.588669 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:16.590440 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:16.591193 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:16.591202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:16.591702 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:32:16.592260 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:16.592277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.593395 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.593399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:16.594661 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:16.594677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:16.595566 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:16.595574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:16.595826 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:16.596522 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:32:16.596546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:32:16.599111 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:32:16.602226 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:16.602236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:16.602256 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:16.602718 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:16.603112 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:16.603118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:16.603122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:16.603335 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:16.603902 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:32:16.603910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:32:16.604298 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 02:32:16.604576 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 02:32:16.605190 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":277,"threads_started":1}
I20260620 02:32:16.608098 16269 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45875
I20260620 02:32:16.608425 16269 heartbeater.cc:464] Registering TS with master...
I20260620 02:32:16.609246 16269 heartbeater.cc:511] Master 127.0.97.62:45875 requested a full tablet report, sending...
I20260620 02:32:16.610848 15946 ts_manager.cc:194] Registered new tserver with Master: 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
I20260620 02:32:16.612190 15946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53159
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:32:16.652813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45875
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:32:16.987571 16275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:16.987864 16275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:16.987962 16275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:16.995946 16275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:16.996115 16275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:16.996310 16275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:32:17.007370 16275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:17.008745 16275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:17.010143 16275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:17.023476 16280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:17.023448 16281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:17.023509 16283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:17.024209 16275 server_base.cc:1034] running on GCE node
I20260620 02:32:17.024871 16275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:17.025991 16275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:17.027231 16275 hybrid_clock.cc:648] HybridClock initialized: now 1781922737027170 us; error 47 us; skew 500 ppm
I20260620 02:32:17.027616 16275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:17 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:17.031116 16275 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:32:17.032810 16275 webserver.cc:460] Webserver started at http://127.0.97.3:35017/ using document root <none> and password file <none>
I20260620 02:32:17.033598 16275 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:17.033756 16275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:17.034104 16275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:32:17.037604 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "bea31de2765b4b0ead4780ca30164deb"
format_stamp: "Formatted at 2026-06-20 02:32:17 on dist-test-slave-r706"
server_key: "84cb08aeaacf552acc885477af42cd01"
server_key_iv: "5eecf940a81d88a1d00e97aa5b781a89"
server_key_version: "encryptionkey@0"
I20260620 02:32:17.038426 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bea31de2765b4b0ead4780ca30164deb"
format_stamp: "Formatted at 2026-06-20 02:32:17 on dist-test-slave-r706"
server_key: "84cb08aeaacf552acc885477af42cd01"
server_key_iv: "5eecf940a81d88a1d00e97aa5b781a89"
server_key_version: "encryptionkey@0"
I20260620 02:32:17.043885 16275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:32:17.047483 16290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:17.048942 16275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 02:32:17.049211 16275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "bea31de2765b4b0ead4780ca30164deb"
format_stamp: "Formatted at 2026-06-20 02:32:17 on dist-test-slave-r706"
server_key: "84cb08aeaacf552acc885477af42cd01"
server_key_iv: "5eecf940a81d88a1d00e97aa5b781a89"
server_key_version: "encryptionkey@0"
I20260620 02:32:17.049439 16275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:17.066673 16275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:17.070958 16275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:17.071276 16275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:17.072324 16275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:32:17.073773 16275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:32:17.073876 16275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:17.074045 16275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:32:17.074126 16275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:17.098719 16135 heartbeater.cc:503] Master 127.0.97.62:45875 was elected leader, sending a full tablet report...
I20260620 02:32:17.108855 16275 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34937
I20260620 02:32:17.108949 16402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34937 every 8 connection(s)
I20260620 02:32:17.110705 16275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:32:17.118947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16275
I20260620 02:32:17.119141 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:32:17.119690 388 external_mini_cluster.cc:1428] Setting key aee1228480e57f00e6a27e5d8568e72b
Jun 20 02:32:17 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:17.132120 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.113346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36647 (local address 127.0.97.62:45875)
0620 02:32:17.113793 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.113798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.114743 (+ 945us) server_negotiation.cc:407] Connection header received
0620 02:32:17.116571 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.116577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.116679 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.116800 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:17.118318 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.119478 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.121518 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.122024 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.122194 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.125955 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:17.125960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:17.125963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:17.125998 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:17.129205 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.129998 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.130002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.130004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.130136 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.130543 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.130547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.130549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.130818 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:17.130929 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.131657 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.131834 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":214,"threads_started":1}
I20260620 02:32:17.133193 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.113884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:36647)
0620 02:32:17.114583 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:17.114619 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:17.116302 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:17.117099 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:17.117108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:17.117606 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:32:17.118147 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.118164 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.119651 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.119655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.121373 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.121387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.122291 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.122296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.122495 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.123310 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:32:17.123349 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:32:17.125728 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:32:17.129366 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:17.129373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:17.129389 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:17.129849 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:17.130253 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:17.130256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:17.130258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:17.130436 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:17.130939 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:32:17.130946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:32:17.131312 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 02:32:17.131543 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 02:32:17.131936 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260620 02:32:17.134718 16403 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45875
I20260620 02:32:17.135047 16403 heartbeater.cc:464] Registering TS with master...
I20260620 02:32:17.135944 16403 heartbeater.cc:511] Master 127.0.97.62:45875 requested a full tablet report, sending...
I20260620 02:32:17.137395 15945 ts_manager.cc:194] Registered new tserver with Master: bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937)
I20260620 02:32:17.138329 15945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36647
I20260620 02:32:17.148321 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:32:17.167271 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.153485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.0.97.62:45875)
0620 02:32:17.153711 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.153720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.154138 (+ 418us) server_negotiation.cc:407] Connection header received
0620 02:32:17.154375 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.154381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.154463 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.154576 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:17.156069 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.156836 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.158259 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.158690 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.158830 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.161638 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:17.161646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:17.161657 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:17.161694 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:17.164406 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.165149 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.165157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.165165 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.165291 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.165708 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.165713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.165715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.166065 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:17.166324 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.166688 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.166931 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:32:17.172098 15945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:32:17.179359 15945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:32:17.203730 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.192157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.0.97.3:34937)
0620 02:32:17.193935 (+ 1778us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.193945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.193979 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:32:17.194079 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.194088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.194349 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.194554 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:17.195427 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.197556 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.201241 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.202506 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.202672 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.202908 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.203195 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.203361 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":1429,"threads_started":1}
I20260620 02:32:17.203873 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.191856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34937 (local address 127.0.0.1:49656)
0620 02:32:17.192577 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:17.192617 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:17.192814 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:17.194578 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:17.194586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:17.194637 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:17.195166 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.195181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.197745 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.197750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.201040 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.201055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.202719 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.202726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.202940 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.203001 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 02:32:17.203136 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 02:32:17.203354 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":36,"thread_start_us":330,"threads_started":1}
I20260620 02:32:17.205282 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.192513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.0.97.2:42077)
0620 02:32:17.193381 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.193393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.194331 (+ 938us) server_negotiation.cc:407] Connection header received
0620 02:32:17.194627 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.194636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.194860 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.195045 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:17.196262 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.198350 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.202652 (+ 4302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.204111 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.204268 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.204467 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.204715 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.204877 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":485,"threads_started":1}
I20260620 02:32:17.206058 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.191950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42077 (local address 127.0.0.1:50718)
0620 02:32:17.194220 (+ 2270us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:17.194238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:17.194436 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:17.195421 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:17.195429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:17.195464 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:17.196019 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.196034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.198581 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.198589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.202478 (+ 3889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.202490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.204335 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.204341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.204520 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.204549 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:32:17.204662 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:32:17.204790 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2159,"mutex_wait_us":1065,"thread_start_us":1987,"threads_started":1}
I20260620 02:32:17.208065 16338 tablet_service.cc:1467] Processing CreateTablet for tablet d73a7e76b8cc4047bbcff51b79c0e281 (DEFAULT_TABLE table=test-table [id=85333c142ec9407580cf405a696539c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:32:17.209717 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73a7e76b8cc4047bbcff51b79c0e281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:17.210263 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.194511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.0.97.1:34655)
0620 02:32:17.201514 (+ 7003us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.201527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.202293 (+ 766us) server_negotiation.cc:407] Connection header received
0620 02:32:17.202404 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.202413 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.202663 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.202852 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:17.203766 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.205936 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.208132 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.209189 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.209336 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.209563 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.209761 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.209929 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6771,"thread_start_us":7435,"threads_started":1}
I20260620 02:32:17.210260 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.193853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:46714)
0620 02:32:17.202023 (+ 8170us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:17.202041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:17.202174 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:17.203118 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:17.203126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:17.203152 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:17.203544 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.203558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.206167 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.206179 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.207965 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.207978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.209387 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.209393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.209559 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.209608 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:32:17.209719 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:32:17.209940 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8029,"thread_start_us":7867,"threads_started":1}
I20260620 02:32:17.209910 16204 tablet_service.cc:1467] Processing CreateTablet for tablet d73a7e76b8cc4047bbcff51b79c0e281 (DEFAULT_TABLE table=test-table [id=85333c142ec9407580cf405a696539c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:32:17.211279 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73a7e76b8cc4047bbcff51b79c0e281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:17.213583 16070 tablet_service.cc:1467] Processing CreateTablet for tablet d73a7e76b8cc4047bbcff51b79c0e281 (DEFAULT_TABLE table=test-table [id=85333c142ec9407580cf405a696539c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:32:17.214691 16070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73a7e76b8cc4047bbcff51b79c0e281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:17.227830 16422 tablet_bootstrap.cc:492] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Bootstrap starting.
I20260620 02:32:17.228751 16424 tablet_bootstrap.cc:492] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Bootstrap starting.
I20260620 02:32:17.232441 16423 tablet_bootstrap.cc:492] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Bootstrap starting.
I20260620 02:32:17.233042 16422 tablet_bootstrap.cc:654] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:17.233263 16424 tablet_bootstrap.cc:654] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:17.234905 16424 log.cc:824] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:17.235222 16422 log.cc:824] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:17.236128 16423 tablet_bootstrap.cc:654] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:17.237601 16423 log.cc:824] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:17.239267 16424 tablet_bootstrap.cc:492] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: No bootstrap required, opened a new log
I20260620 02:32:17.239291 16422 tablet_bootstrap.cc:492] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: No bootstrap required, opened a new log
I20260620 02:32:17.239583 16424 ts_tablet_manager.cc:1397] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260620 02:32:17.239603 16422 ts_tablet_manager.cc:1397] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 02:32:17.240904 16423 tablet_bootstrap.cc:492] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: No bootstrap required, opened a new log
I20260620 02:32:17.241245 16423 ts_tablet_manager.cc:1397] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:32:17.248245 16424 raft_consensus.cc:348] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.248714 16424 raft_consensus.cc:374] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:17.248822 16424 raft_consensus.cc:729] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59be24c5e994c2988f086fe7eb2ed86, State: Initialized, Role: FOLLOWER
I20260620 02:32:17.249555 16424 consensus_queue.cc:260] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.250599 16422 raft_consensus.cc:348] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.251130 16422 raft_consensus.cc:374] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:17.251219 16422 raft_consensus.cc:729] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7640c157bb43818ad68268ebee8719, State: Initialized, Role: FOLLOWER
I20260620 02:32:17.251858 16424 ts_tablet_manager.cc:1428] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260620 02:32:17.252068 16422 consensus_queue.cc:260] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.252242 16423 raft_consensus.cc:348] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.252686 16423 raft_consensus.cc:374] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:17.252808 16423 raft_consensus.cc:729] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea31de2765b4b0ead4780ca30164deb, State: Initialized, Role: FOLLOWER
I20260620 02:32:17.253602 16269 heartbeater.cc:503] Master 127.0.97.62:45875 was elected leader, sending a full tablet report...
I20260620 02:32:17.254330 16422 ts_tablet_manager.cc:1428] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260620 02:32:17.254885 16423 consensus_queue.cc:260] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.257239 16403 heartbeater.cc:503] Master 127.0.97.62:45875 was elected leader, sending a full tablet report...
I20260620 02:32:17.257913 16423 ts_tablet_manager.cc:1428] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260620 02:32:17.315807 16136 tablet.cc:2369] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:32:17.337308 16270 tablet.cc:2369] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:32:17.340032 16430 raft_consensus.cc:482] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:32:17.340340 16430 raft_consensus.cc:504] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.342659 16430 leader_election.cc:288] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077), e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1:34655)
I20260620 02:32:17.349887 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.343033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39783 (local address 127.0.97.2:42077)
0620 02:32:17.343244 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.343251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.343320 (+ 69us) server_negotiation.cc:407] Connection header received
0620 02:32:17.343637 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.343644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.343727 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.343821 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:17.344443 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.345480 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.347840 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.348979 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.349147 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.349247 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.349485 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.349625 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:32:17.349915 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.342901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42077 (local address 127.0.97.3:39783)
0620 02:32:17.343241 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:17.343264 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:17.343398 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:17.344007 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:17.344024 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:17.344095 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:17.344284 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.344293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.345612 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.345616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.347683 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.347703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.349147 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.349151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.349275 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.349345 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 02:32:17.349439 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:32:17.349575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":19,"spinlock_wait_cycles":20224}
I20260620 02:32:17.350967 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.343211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47431 (local address 127.0.97.1:34655)
0620 02:32:17.343468 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.343479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.344040 (+ 561us) server_negotiation.cc:407] Connection header received
0620 02:32:17.344271 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.344275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.344369 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.344481 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:17.344923 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.345870 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.348153 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.349386 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.349532 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.349625 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.350593 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.350730 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 02:32:17.351030 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73a7e76b8cc4047bbcff51b79c0e281" candidate_uuid: "bea31de2765b4b0ead4780ca30164deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7640c157bb43818ad68268ebee8719" is_pre_election: true
I20260620 02:32:17.351330 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.343128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.97.3:47431)
0620 02:32:17.343920 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:17.343937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:17.344052 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:17.344498 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:17.344500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:17.344527 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:17.344781 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.344788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.346013 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.346018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.347996 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:17.348006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.349770 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.349776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:17.350420 (+ 644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.350457 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:32:17.350560 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:32:17.350687 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":27,"thread_start_us":456,"threads_started":1}
I20260620 02:32:17.351658 16224 raft_consensus.cc:2454] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bea31de2765b4b0ead4780ca30164deb in term 0.
I20260620 02:32:17.352345 16291 leader_election.cc:302] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7640c157bb43818ad68268ebee8719, bea31de2765b4b0ead4780ca30164deb; no voters:
I20260620 02:32:17.352478 16090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73a7e76b8cc4047bbcff51b79c0e281" candidate_uuid: "bea31de2765b4b0ead4780ca30164deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59be24c5e994c2988f086fe7eb2ed86" is_pre_election: true
I20260620 02:32:17.352754 16430 raft_consensus.cc:2781] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:32:17.352911 16430 raft_consensus.cc:482] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:32:17.353049 16430 raft_consensus.cc:3037] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:17.353094 16090 raft_consensus.cc:2454] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bea31de2765b4b0ead4780ca30164deb in term 0.
I20260620 02:32:17.355520 16430 raft_consensus.cc:504] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
I20260620 02:32:17.356138 16430 leader_election.cc:288] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [CANDIDATE]: Term 1 election: Requested vote from peers 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077), e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1:34655)
I20260620 02:32:17.356616 16090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73a7e76b8cc4047bbcff51b79c0e281" candidate_uuid: "bea31de2765b4b0ead4780ca30164deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59be24c5e994c2988f086fe7eb2ed86"
I20260620 02:32:17.356618 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73a7e76b8cc4047bbcff51b79c0e281" candidate_uuid: "bea31de2765b4b0ead4780ca30164deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7640c157bb43818ad68268ebee8719"
I20260620 02:32:17.356818 16090 raft_consensus.cc:3037] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:17.356842 16224 raft_consensus.cc:3037] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:17.359077 16090 raft_consensus.cc:2454] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea31de2765b4b0ead4780ca30164deb in term 1.
I20260620 02:32:17.359119 16224 raft_consensus.cc:2454] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea31de2765b4b0ead4780ca30164deb in term 1.
I20260620 02:32:17.359545 16293 leader_election.cc:302] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bea31de2765b4b0ead4780ca30164deb, e59be24c5e994c2988f086fe7eb2ed86; no voters:
I20260620 02:32:17.359913 16430 raft_consensus.cc:2781] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:32:17.360378 16430 raft_consensus.cc:686] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 1 LEADER]: Becoming Leader. State: Replica: bea31de2765b4b0ead4780ca30164deb, State: Running, Role: LEADER
I20260620 02:32:17.360934 16430 consensus_queue.cc:237] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } }
W20260620 02:32:17.362278 16404 tablet.cc:2369] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:32:17.365696 15945 catalog_manager.cc:5495] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb reported cstate change: term changed from 0 to 1, leader changed from <none> to bea31de2765b4b0ead4780ca30164deb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bea31de2765b4b0ead4780ca30164deb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } health_report { overall_health: UNKNOWN } } }
I20260620 02:32:17.400221 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.395024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.97.3:34937)
0620 02:32:17.395234 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.395238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.395258 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:32:17.395361 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.395364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.395441 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.395583 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:32:17.395937 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.396683 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.398220 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.398609 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.398707 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.399144 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:32:17.399655 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:32:17.399769 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.399848 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.399912 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:32:17.408352 16224 raft_consensus.cc:1261] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Refusing update from remote peer bea31de2765b4b0ead4780ca30164deb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:32:17.408360 16090 raft_consensus.cc:1261] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Refusing update from remote peer bea31de2765b4b0ead4780ca30164deb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:32:17.409372 16432 consensus_queue.cc:1035] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:32:17.409749 16430 consensus_queue.cc:1035] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:32:17.424387 16435 mvcc.cc:205] Tried to move back new op lower bound from 7298755532415102976 to 7298755532234936320. Current Snapshot: MvccSnapshot[applied={T|T < 7298755532415102976}]
I20260620 02:32:17.427358 16436 mvcc.cc:205] Tried to move back new op lower bound from 7298755532415102976 to 7298755532234936320. Current Snapshot: MvccSnapshot[applied={T|T < 7298755532415102976}]
I20260620 02:32:17.454277 15946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:32:17.454556 15946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:32:17.457919 15946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Sending DeleteTablet for 3 replicas of tablet d73a7e76b8cc4047bbcff51b79c0e281
I20260620 02:32:17.459074 16338 tablet_service.cc:1514] Processing DeleteTablet for tablet d73a7e76b8cc4047bbcff51b79c0e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49656
I20260620 02:32:17.459105 16204 tablet_service.cc:1514] Processing DeleteTablet for tablet d73a7e76b8cc4047bbcff51b79c0e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50718
I20260620 02:32:17.459125 16070 tablet_service.cc:1514] Processing DeleteTablet for tablet d73a7e76b8cc4047bbcff51b79c0e281 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46714
I20260620 02:32:17.462301 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16007
I20260620 02:32:17.473583 16446 tablet_replica.cc:330] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: stopping tablet replica
I20260620 02:32:17.474233 16446 raft_consensus.cc:2229] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:32:17.474951 16446 raft_consensus.cc:2258] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:32:17.475827 16445 tablet_replica.cc:330] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: stopping tablet replica
I20260620 02:32:17.476200 16445 raft_consensus.cc:2229] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:32:17.476738 16445 raft_consensus.cc:2258] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:32:17.477782 16444 tablet_replica.cc:330] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: stopping tablet replica
I20260620 02:32:17.478185 16444 raft_consensus.cc:2229] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:32:17.478983 16444 raft_consensus.cc:2258] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:32:17.479192 16445 ts_tablet_manager.cc:1905] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:32:17.480554 16446 ts_tablet_manager.cc:1905] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:32:17.486403 16445 ts_tablet_manager.cc:1918] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:32:17.486693 16445 log.cc:1196] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/d73a7e76b8cc4047bbcff51b79c0e281
I20260620 02:32:17.487305 16445 ts_tablet_manager.cc:1939] T d73a7e76b8cc4047bbcff51b79c0e281 P 8d7640c157bb43818ad68268ebee8719: Deleting consensus metadata
I20260620 02:32:17.487802 16446 ts_tablet_manager.cc:1918] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:32:17.487989 16446 log.cc:1196] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/d73a7e76b8cc4047bbcff51b79c0e281
I20260620 02:32:17.488580 16446 ts_tablet_manager.cc:1939] T d73a7e76b8cc4047bbcff51b79c0e281 P e59be24c5e994c2988f086fe7eb2ed86: Deleting consensus metadata
I20260620 02:32:17.489675 16444 ts_tablet_manager.cc:1905] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:32:17.490156 15930 catalog_manager.cc:4841] TS 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077): tablet d73a7e76b8cc4047bbcff51b79c0e281 (table test-table [id=85333c142ec9407580cf405a696539c6]) successfully deleted
I20260620 02:32:17.490460 15932 catalog_manager.cc:4841] TS e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1:34655): tablet d73a7e76b8cc4047bbcff51b79c0e281 (table test-table [id=85333c142ec9407580cf405a696539c6]) successfully deleted
I20260620 02:32:17.495689 16444 ts_tablet_manager.cc:1918] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:32:17.495899 16444 log.cc:1196] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/d73a7e76b8cc4047bbcff51b79c0e281
I20260620 02:32:17.496546 16444 ts_tablet_manager.cc:1939] T d73a7e76b8cc4047bbcff51b79c0e281 P bea31de2765b4b0ead4780ca30164deb: Deleting consensus metadata
Jun 20 02:32:17 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:32:17.498342 15933 catalog_manager.cc:4841] TS bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937): tablet d73a7e76b8cc4047bbcff51b79c0e281 (table test-table [id=85333c142ec9407580cf405a696539c6]) successfully deleted
I20260620 02:32:17.505909 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.463787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.0.97.1:34655)
0620 02:32:17.463977 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.463983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.481624 (+ 17641us) server_negotiation.cc:407] Connection header received
0620 02:32:17.482146 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.482154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.482251 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.482342 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:17.484450 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.485704 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.487821 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.488534 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.488685 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.499192 (+ 10507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:17.499208 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:17.499224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:17.499264 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:17.502408 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.503431 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.503438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.503441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.503582 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.504004 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.504007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.504009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.504389 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:17.504722 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.505190 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.505440 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:32:17.506726 16130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:17.606853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16007
I20260620 02:32:17.628425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16141
Jun 20 02:32:17 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:32:17.647133 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.629372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.0.97.2:42077)
0620 02:32:17.629620 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.629626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.629807 (+ 181us) server_negotiation.cc:407] Connection header received
0620 02:32:17.629959 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.629963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.630042 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.630151 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:17.631969 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.632734 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.634887 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.635280 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.635382 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.641092 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:17.641110 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:17.641131 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:17.641172 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:17.644115 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.644967 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.644975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.644978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.645238 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.645617 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.645620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.645622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.646028 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:17.646253 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.646624 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.646788 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:32:17.647907 16264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:17.739151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16141
I20260620 02:32:17.757768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16275
Jun 20 02:32:17 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:32:17.778079 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:17.758818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.0.97.3:34937)
0620 02:32:17.758988 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:17.758993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:17.759010 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:32:17.759087 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:17.759091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:17.759153 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:17.759235 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:17.761112 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.761938 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.763894 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:17.764324 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:17.764419 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:17.770489 (+ 6070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:17.770504 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:17.770523 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:17.770567 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:17.774584 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.775395 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.775405 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.775409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.775593 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:17.776051 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:17.776056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:17.776074 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:17.776623 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:17.776980 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:17.777396 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 02:32:17.777646 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:32:17.778983 16398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:17.902666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16275
I20260620 02:32:17.919792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 15914
I20260620 02:32:17.920975 15976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:18.035857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 15914
I20260620 02:32:38.060243 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45875
--webserver_interface=127.0.97.62
--webserver_port=35685
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:32:38.418689 16452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:38.418994 16452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:38.419178 16452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:38.427328 16452 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:32:38.427495 16452 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:32:38.427569 16452 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:32:38.427647 16452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:32:38.427731 16452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:38.427836 16452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:38.427927 16452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:32:38.428011 16452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:32:38.438561 16452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:38.439951 16452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:38.441385 16452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:38.466988 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:38.467026 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:38.467022 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:38.467865 16452 server_base.cc:1034] running on GCE node
I20260620 02:32:38.468907 16452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:38.470691 16452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:38.471939 16452 hybrid_clock.cc:648] HybridClock initialized: now 1781922758471866 us; error 52 us; skew 500 ppm
I20260620 02:32:38.472330 16452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:38 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:38.480434 16452 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:32:38.482422 16452 webserver.cc:460] Webserver started at http://127.0.97.62:35685/ using document root <none> and password file <none>
I20260620 02:32:38.483325 16452 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:38.483480 16452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:38.490343 16452 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:32:38.493762 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:38.495463 16452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:32:38.495705 16452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "61f8392a77a7401b905389746eb42a7e"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "90137b48f52bef734064b580c894ffc8"
server_key_iv: "3b7a9bd35320ceb6f1c7153deef463bd"
server_key_version: "encryptionkey@0"
I20260620 02:32:38.496846 16452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:38.525238 16452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:38.530040 16452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:38.530405 16452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:38.551149 16452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45875
I20260620 02:32:38.551208 16519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45875 every 8 connection(s)
I20260620 02:32:38.552879 16452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:32:38.560313 16520 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:32:38.560349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16452
I20260620 02:32:38.561267 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34655
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45875
--webserver_port=43575
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:32:38.563925 16520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap starting.
I20260620 02:32:38.584497 16520 log.cc:824] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:38.597306 16520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:32:38.597929 16520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap complete.
I20260620 02:32:38.608112 16520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:38.609112 16520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8392a77a7401b905389746eb42a7e, State: Initialized, Role: FOLLOWER
I20260620 02:32:38.609822 16520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:38.610098 16520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:32:38.610244 16520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:32:38.610428 16520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:32:38.614557 16520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:38.615317 16520 leader_election.cc:302] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f8392a77a7401b905389746eb42a7e; no voters:
I20260620 02:32:38.616098 16520 leader_election.cc:288] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:32:38.616241 16524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:32:38.617774 16524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 LEADER]: Becoming Leader. State: Replica: 61f8392a77a7401b905389746eb42a7e, State: Running, Role: LEADER
I20260620 02:32:38.618480 16524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:32:38.618840 16520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:32:38.621225 16526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f8392a77a7401b905389746eb42a7e. Latest consensus state: current_term: 2 leader_uuid: "61f8392a77a7401b905389746eb42a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } } }
I20260620 02:32:38.621675 16526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: This master's current role is: LEADER
I20260620 02:32:38.622781 16525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "61f8392a77a7401b905389746eb42a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } } }
I20260620 02:32:38.623162 16525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: This master's current role is: LEADER
I20260620 02:32:38.638649 16529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:32:38.648012 16529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:32:38.649748 16529 catalog_manager.cc:1259] Loaded cluster ID: 4e69bc904c3340ab860474d6bf69f6d5
I20260620 02:32:38.649878 16529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:32:38.655005 16529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:32:38.657202 16529 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Loaded TSK: 0
I20260620 02:32:38.690814 16529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Generated new TSK 1
I20260620 02:32:38.691275 16529 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:32:38.948783 16522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:38.949175 16522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:38.949358 16522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:38.960062 16522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:38.960230 16522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:38.960449 16522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:32:38.971434 16522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43575
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:38.972891 16522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:38.974370 16522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:38.989681 16547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:38.989670 16548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:38.990634 16522 server_base.cc:1034] running on GCE node
W20260620 02:32:38.993412 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:38.994350 16522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:38.995637 16522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:38.996930 16522 hybrid_clock.cc:648] HybridClock initialized: now 1781922758996866 us; error 54 us; skew 500 ppm
I20260620 02:32:38.997371 16522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:38 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:39.001473 16522 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:32:39.003562 16522 webserver.cc:460] Webserver started at http://127.0.97.1:43575/ using document root <none> and password file <none>
I20260620 02:32:39.004376 16522 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:39.004565 16522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:39.011121 16522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:32:39.014683 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.016976 16522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:32:39.017290 16522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "e59be24c5e994c2988f086fe7eb2ed86"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "f0b019b5b5789d8e7b1c18a7b1b3ffdc"
server_key_iv: "0c2b623a63069e5b708a0799e589ecbd"
server_key_version: "encryptionkey@0"
I20260620 02:32:39.018759 16522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:39.038460 16522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:39.043438 16522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:39.043759 16522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:39.044888 16522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:32:39.046402 16522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:32:39.046514 16522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.046636 16522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:32:39.046723 16522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.084802 16522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34655
I20260620 02:32:39.084865 16669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34655 every 8 connection(s)
I20260620 02:32:39.086401 16522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:32:39.090412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16522
I20260620 02:32:39.091266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42077
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45875
--webserver_port=39289
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:32:39 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:39.116158 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:39.089867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49017 (local address 127.0.97.62:45875)
0620 02:32:39.091168 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:39.091191 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:39.093903 (+ 2712us) server_negotiation.cc:407] Connection header received
0620 02:32:39.096310 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:39.096324 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:39.096702 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:39.097167 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:39.098879 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.100197 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.102036 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.102784 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.103004 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:39.107443 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:39.107452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:39.107468 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:39.107511 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:39.110924 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:39.112159 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:39.112167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:39.112173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:39.112365 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:39.112945 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:39.112949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:39.112951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:39.113541 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:39.113715 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:39.114816 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0620 02:32:39.115175 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":1782,"threads_started":1}
I20260620 02:32:39.116717 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:39.090016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:49017)
0620 02:32:39.093690 (+ 3674us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:39.093751 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:39.095675 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:39.097371 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:39.097385 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:39.098034 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:32:39.098593 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:39.098611 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.100407 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.100414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:39.101878 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:39.101892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.103044 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.103052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:39.103341 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:39.104086 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:32:39.104109 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:32:39.107066 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:32:39.111154 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:39.111168 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:39.111190 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:39.111959 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:39.112499 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:39.112507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:39.112512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:39.112801 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:39.113690 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:32:39.113703 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:32:39.114216 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 02:32:39.114565 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 02:32:39.115265 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3355,"thread_start_us":3116,"threads_started":1}
I20260620 02:32:39.120167 16670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45875
I20260620 02:32:39.120574 16670 heartbeater.cc:464] Registering TS with master...
I20260620 02:32:39.121414 16670 heartbeater.cc:511] Master 127.0.97.62:45875 requested a full tablet report, sending...
I20260620 02:32:39.123972 16485 ts_manager.cc:194] Registered new tserver with Master: e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1:34655)
I20260620 02:32:39.126227 16485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49017
W20260620 02:32:39.454952 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:39.455274 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:39.455374 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:39.463325 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:39.463451 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:39.463567 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:32:39.473953 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39289
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:39.475313 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:39.476648 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:39.490844 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:39.491009 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:39.491480 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:39.491899 16674 server_base.cc:1034] running on GCE node
I20260620 02:32:39.492596 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:39.493670 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:39.494906 16674 hybrid_clock.cc:648] HybridClock initialized: now 1781922759494859 us; error 44 us; skew 500 ppm
I20260620 02:32:39.495236 16674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:39 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:39.498734 16674 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:32:39.500310 16674 webserver.cc:460] Webserver started at http://127.0.97.2:39289/ using document root <none> and password file <none>
I20260620 02:32:39.501093 16674 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:39.501267 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:39.506867 16674 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:32:39.509785 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.511211 16674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:32:39.511399 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "8d7640c157bb43818ad68268ebee8719"
format_stamp: "Formatted at 2026-06-20 02:32:16 on dist-test-slave-r706"
server_key: "d1d7c822960c382aaf46499e74f4887d"
server_key_iv: "47ec786077985440b34c995b083bce48"
server_key_version: "encryptionkey@0"
I20260620 02:32:39.512415 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:39.526930 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:39.531424 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:39.531723 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:39.532754 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:32:39.534240 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:32:39.534371 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.534507 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:32:39.534595 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.570415 16674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42077
I20260620 02:32:39.570461 16801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42077 every 8 connection(s)
I20260620 02:32:39.571913 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:32:39.580638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16674
I20260620 02:32:39.581507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34937
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45875
--webserver_port=35017
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:32:39 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:39.597397 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:39.574663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36649 (local address 127.0.97.62:45875)
0620 02:32:39.574881 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:39.574886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:39.575987 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 02:32:39.578033 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:39.578038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:39.578168 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:39.578291 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:39.579888 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.580681 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.583514 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.584131 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.584265 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:39.590596 (+ 6331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:39.590602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:39.590605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:39.590645 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:39.593676 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:39.594779 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:39.594784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:39.594786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:39.594934 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:39.595502 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:39.595505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:39.595507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:39.595802 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:39.595919 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:39.596875 (+ 956us) server_negotiation.cc:299] Negotiation successful
0620 02:32:39.597132 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:32:39.598941 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:39.575029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:36649)
0620 02:32:39.575791 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:39.575836 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:39.577733 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:39.578535 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:39.578544 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:39.579092 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:32:39.579704 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:39.579723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.580916 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.580922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:39.583273 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:39.583304 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:39.584450 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:39.584461 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:39.584967 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:39.585882 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:32:39.585919 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:32:39.590265 (+ 4346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:32:39.593860 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:39.593874 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:39.593899 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:39.594605 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:39.595057 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:39.595064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:39.595071 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:39.595364 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:39.595932 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:32:39.595943 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:32:39.596438 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 02:32:39.596791 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 02:32:39.597439 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":317,"threads_started":1}
I20260620 02:32:39.600601 16802 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45875
I20260620 02:32:39.600980 16802 heartbeater.cc:464] Registering TS with master...
I20260620 02:32:39.601946 16802 heartbeater.cc:511] Master 127.0.97.62:45875 requested a full tablet report, sending...
I20260620 02:32:39.603581 16485 ts_manager.cc:194] Registered new tserver with Master: 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
I20260620 02:32:39.604398 16485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36649
W20260620 02:32:39.937923 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:32:39.938270 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:32:39.938443 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:32:39.945895 16805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:32:39.946065 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:32:39.946259 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:32:39.956516 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35017
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:32:39.958010 16805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:32:39.959334 16805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:32:39.972631 16810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:39.972571 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:32:39.972661 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:32:39.973726 16805 server_base.cc:1034] running on GCE node
I20260620 02:32:39.974376 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:32:39.975370 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:32:39.976635 16805 hybrid_clock.cc:648] HybridClock initialized: now 1781922759976590 us; error 60 us; skew 500 ppm
I20260620 02:32:39.976980 16805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:32:39 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:32:39.980242 16805 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:32:39.981758 16805 webserver.cc:460] Webserver started at http://127.0.97.3:35017/ using document root <none> and password file <none>
I20260620 02:32:39.982445 16805 fs_manager.cc:359] Metadata directory not provided
I20260620 02:32:39.982566 16805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:32:39.988489 16805 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:32:39.991600 16820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:39.993094 16805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:32:39.993252 16805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "bea31de2765b4b0ead4780ca30164deb"
format_stamp: "Formatted at 2026-06-20 02:32:17 on dist-test-slave-r706"
server_key: "84cb08aeaacf552acc885477af42cd01"
server_key_iv: "5eecf940a81d88a1d00e97aa5b781a89"
server_key_version: "encryptionkey@0"
I20260620 02:32:39.994318 16805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:32:40.033785 16805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:32:40.038434 16805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:32:40.038676 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:32:40.039708 16805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:32:40.041114 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:32:40.041209 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:40.041289 16805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:32:40.041347 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:32:40.077742 16805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34937
I20260620 02:32:40.077838 16932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34937 every 8 connection(s)
I20260620 02:32:40.079306 16805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:32:40.085999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16805
Jun 20 02:32:40 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:32:40.108603 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.082301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42133 (local address 127.0.97.62:45875)
0620 02:32:40.082501 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.082506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.083782 (+ 1276us) server_negotiation.cc:407] Connection header received
0620 02:32:40.085891 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.085899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.085991 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.086074 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:40.088730 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.090392 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.093884 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.094920 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.095065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.100955 (+ 5890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:40.100960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:40.100964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:40.101064 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:40.104757 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.105683 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.105689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.105693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.105831 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.106292 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.106302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.106304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.106578 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:40.106745 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.107650 (+ 905us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.108075 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:32:40.110085 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.082813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:42133)
0620 02:32:40.083590 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.083629 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:40.085526 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:40.086563 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:40.086571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:40.087432 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:32:40.088443 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.088476 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.090743 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.090752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.093601 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.093628 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.095567 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.095590 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.096055 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.097161 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:32:40.097215 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:32:40.100682 (+ 3467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:32:40.104923 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:40.104934 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:40.104961 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:40.105523 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:40.105948 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:32:40.105954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:32:40.105958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:32:40.106166 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:32:40.106735 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:32:40.106743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:32:40.107163 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 02:32:40.107526 (+ 363us) client_negotiation.cc:240] Negotiation successful
0620 02:32:40.108405 (+ 879us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":319,"threads_started":1}
I20260620 02:32:40.111931 16933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45875
I20260620 02:32:40.112278 16933 heartbeater.cc:464] Registering TS with master...
I20260620 02:32:40.113113 16933 heartbeater.cc:511] Master 127.0.97.62:45875 requested a full tablet report, sending...
I20260620 02:32:40.114787 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.087070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.0.97.62:45875)
0620 02:32:40.097092 (+ 10022us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.097105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.097642 (+ 537us) server_negotiation.cc:407] Connection header received
0620 02:32:40.097891 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.097895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.097982 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.098100 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:40.100087 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.101289 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.103472 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.103979 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.104084 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.107270 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:40.107276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:40.107281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:40.107315 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:40.111383 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.112412 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.112421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.112425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.112618 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.113088 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.113093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.113097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.113442 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:40.113721 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.114130 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.114374 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9834,"thread_start_us":9708,"threads_started":1}
I20260620 02:32:40.114759 16485 ts_manager.cc:194] Registered new tserver with Master: bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937)
I20260620 02:32:40.116041 16485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42133
I20260620 02:32:40.117797 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:32:40.125118 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.119145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.0.97.62:45875)
0620 02:32:40.119407 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.119414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.119759 (+ 345us) server_negotiation.cc:407] Connection header received
0620 02:32:40.119967 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.119971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.120112 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.120243 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:32:40.120663 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.121464 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.122998 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.123410 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.123517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.124266 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:32:40.124490 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 02:32:40.124819 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39472: token expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 02:32:40.125269 16936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39472: token expired
W20260620 02:32:40.125913 16407 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:45875)
I20260620 02:32:40.130057 16670 heartbeater.cc:503] Master 127.0.97.62:45875 was elected leader, sending a full tablet report...
I20260620 02:32:40.141314 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.126410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.0.97.62:45875)
0620 02:32:40.126599 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.126604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.126623 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:32:40.126831 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.126837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.126911 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.127016 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:40.128545 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.129401 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.131193 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.131665 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.131814 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.135130 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:40.135136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:40.135139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:40.135183 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:40.138237 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.139141 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.139146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.139148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.139316 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.139721 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.139725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.139727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.140125 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:40.140331 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.140688 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.140888 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:32:40.145102 16484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:32:40.153544 16484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:32:40.179539 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.166640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.0.97.1:34655)
0620 02:32:40.169807 (+ 3167us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.169817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.169844 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:32:40.169934 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.169941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.170113 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.170522 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:40.171437 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.173296 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.177143 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.178202 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.178378 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.178639 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.178973 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.179152 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2981,"thread_start_us":2807,"threads_started":1}
I20260620 02:32:40.179783 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.167650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34937 (local address 127.0.0.1:55546)
0620 02:32:40.169050 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.169070 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:40.169192 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:40.170370 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:40.170378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:40.170423 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:40.170981 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.171005 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.173544 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.173552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.176622 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.176640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.178753 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.178760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.178988 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.179019 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:32:40.179073 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 02:32:40.179322 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"mutex_wait_us":91,"spinlock_wait_cycles":384,"thread_start_us":1101,"threads_started":1}
I20260620 02:32:40.180397 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.166342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:36658)
0620 02:32:40.168304 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.168360 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:40.168545 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:40.170370 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:40.170378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:40.170430 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:40.171237 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.171248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.173504 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.173514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.176936 (+ 3422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.176960 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.178458 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.178464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.178714 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.178760 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:32:40.179196 (+ 436us) client_negotiation.cc:240] Negotiation successful
0620 02:32:40.180084 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"mutex_wait_us":1819,"spinlock_wait_cycles":7296,"thread_start_us":1445,"threads_started":1}
I20260620 02:32:40.181558 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.168542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.97.3:34937)
0620 02:32:40.169357 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.169369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.169407 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:32:40.169520 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.169530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.170084 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.171250 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:40.171614 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.173396 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.176877 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.178492 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.180671 (+ 2179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.180918 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.181000 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.181216 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":447,"threads_started":1}
I20260620 02:32:40.183996 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.170190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42077 (local address 127.0.0.1:32842)
0620 02:32:40.175476 (+ 5286us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.175502 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:40.175869 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:40.176507 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:40.176512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:40.176538 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:40.177041 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.177058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.179317 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.179323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.181369 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.181384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.183195 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.183202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.183382 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.183415 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:32:40.183513 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:32:40.183652 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5151,"thread_start_us":4405,"threads_started":1}
I20260620 02:32:40.184129 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.171160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.97.2:42077)
0620 02:32:40.174418 (+ 3258us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.174428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.175707 (+ 1279us) server_negotiation.cc:407] Connection header received
0620 02:32:40.176062 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.176069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.176286 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.176504 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:40.177245 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.178792 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.181549 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.182924 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.183087 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.183279 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.183615 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.183799 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3073,"thread_start_us":2909,"threads_started":1}
I20260620 02:32:40.185175 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 1f890d08f474456c87facbe35d926df5 (DEFAULT_TABLE table=test-table [id=f6dc1be1e9f4497b88351c2a8308c4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:32:40.185233 16868 tablet_service.cc:1467] Processing CreateTablet for tablet 1f890d08f474456c87facbe35d926df5 (DEFAULT_TABLE table=test-table [id=f6dc1be1e9f4497b88351c2a8308c4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:32:40.186793 16868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f890d08f474456c87facbe35d926df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:40.186812 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f890d08f474456c87facbe35d926df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:40.191320 16737 tablet_service.cc:1467] Processing CreateTablet for tablet 1f890d08f474456c87facbe35d926df5 (DEFAULT_TABLE table=test-table [id=f6dc1be1e9f4497b88351c2a8308c4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:32:40.192857 16737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f890d08f474456c87facbe35d926df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:32:40.199559 16949 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Bootstrap starting.
I20260620 02:32:40.203208 16949 tablet_bootstrap.cc:654] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:40.204348 16949 log.cc:824] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:40.207616 16949 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: No bootstrap required, opened a new log
I20260620 02:32:40.207974 16949 ts_tablet_manager.cc:1397] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 02:32:40.213613 16952 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Bootstrap starting.
I20260620 02:32:40.215080 16951 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Bootstrap starting.
I20260620 02:32:40.217933 16952 tablet_bootstrap.cc:654] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:40.218920 16951 tablet_bootstrap.cc:654] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Neither blocks nor log segments found. Creating new log.
I20260620 02:32:40.219306 16952 log.cc:824] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:40.219002 16949 raft_consensus.cc:348] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.219587 16949 raft_consensus.cc:374] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:40.219705 16949 raft_consensus.cc:729] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59be24c5e994c2988f086fe7eb2ed86, State: Initialized, Role: FOLLOWER
I20260620 02:32:40.220569 16949 consensus_queue.cc:260] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.220892 16951 log.cc:824] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:32:40.223258 16949 ts_tablet_manager.cc:1428] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260620 02:32:40.225862 16952 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: No bootstrap required, opened a new log
I20260620 02:32:40.226223 16952 ts_tablet_manager.cc:1397] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260620 02:32:40.226289 16951 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: No bootstrap required, opened a new log
I20260620 02:32:40.226650 16951 ts_tablet_manager.cc:1397] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 02:32:40.234925 16952 raft_consensus.cc:348] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.234954 16951 raft_consensus.cc:348] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.235368 16952 raft_consensus.cc:374] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:40.235378 16951 raft_consensus.cc:374] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:32:40.235486 16952 raft_consensus.cc:729] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7640c157bb43818ad68268ebee8719, State: Initialized, Role: FOLLOWER
I20260620 02:32:40.235507 16951 raft_consensus.cc:729] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea31de2765b4b0ead4780ca30164deb, State: Initialized, Role: FOLLOWER
I20260620 02:32:40.236161 16951 consensus_queue.cc:260] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.236160 16952 consensus_queue.cc:260] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.237528 16933 heartbeater.cc:503] Master 127.0.97.62:45875 was elected leader, sending a full tablet report...
I20260620 02:32:40.238009 16802 heartbeater.cc:503] Master 127.0.97.62:45875 was elected leader, sending a full tablet report...
I20260620 02:32:40.238135 16951 ts_tablet_manager.cc:1428] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 02:32:40.238619 16952 ts_tablet_manager.cc:1428] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260620 02:32:40.323992 16803 tablet.cc:2369] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:32:40.331110 16934 tablet.cc:2369] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:32:40.338948 16671 tablet.cc:2369] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:32:40.345677 16954 raft_consensus.cc:482] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:32:40.346017 16954 raft_consensus.cc:504] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.347754 16954 leader_election.cc:288] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937), 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
I20260620 02:32:40.355485 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.348206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37397 (local address 127.0.97.2:42077)
0620 02:32:40.348459 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.348467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.349178 (+ 711us) server_negotiation.cc:407] Connection header received
0620 02:32:40.349335 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.349345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.349491 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.349612 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:40.350727 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.351577 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.353524 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.354356 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.354529 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.354651 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.355045 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.355202 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:32:40.355479 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.348248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60379 (local address 127.0.97.3:34937)
0620 02:32:40.348623 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.348629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.349063 (+ 434us) server_negotiation.cc:407] Connection header received
0620 02:32:40.349377 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.349383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.349489 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.349556 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:32:40.350655 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.351496 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.353506 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.354347 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.354523 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.354650 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.355051 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.355195 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260620 02:32:40.355667 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.348187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34937 (local address 127.0.97.1:60379)
0620 02:32:40.348941 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.348960 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:40.349167 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:40.349759 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:40.349766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:40.349784 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:40.350475 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.350489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.351648 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.351654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.353327 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.353343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.354628 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.354636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.354844 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.354900 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:32:40.355010 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:32:40.355185 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":87,"spinlock_wait_cycles":21760,"thread_start_us":480,"threads_started":1}
I20260620 02:32:40.355832 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.348079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42077 (local address 127.0.97.1:37397)
0620 02:32:40.348898 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.348920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:32:40.349139 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:32:40.349648 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:32:40.349653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:32:40.349729 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:32:40.350565 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.350577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.351750 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.351754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.353358 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:32:40.353372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.354675 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.354683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:32:40.354849 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.354892 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:32:40.355013 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 02:32:40.355203 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":58,"thread_start_us":385,"threads_started":1}
I20260620 02:32:40.356832 16757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f890d08f474456c87facbe35d926df5" candidate_uuid: "e59be24c5e994c2988f086fe7eb2ed86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7640c157bb43818ad68268ebee8719" is_pre_election: true
I20260620 02:32:40.356827 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f890d08f474456c87facbe35d926df5" candidate_uuid: "e59be24c5e994c2988f086fe7eb2ed86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea31de2765b4b0ead4780ca30164deb" is_pre_election: true
I20260620 02:32:40.357399 16888 raft_consensus.cc:2454] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59be24c5e994c2988f086fe7eb2ed86 in term 0.
I20260620 02:32:40.357399 16757 raft_consensus.cc:2454] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59be24c5e994c2988f086fe7eb2ed86 in term 0.
I20260620 02:32:40.358247 16558 leader_election.cc:302] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d7640c157bb43818ad68268ebee8719, bea31de2765b4b0ead4780ca30164deb, e59be24c5e994c2988f086fe7eb2ed86; no voters:
I20260620 02:32:40.358693 16954 raft_consensus.cc:2781] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:32:40.358865 16954 raft_consensus.cc:482] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:32:40.358982 16954 raft_consensus.cc:3037] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:40.361308 16954 raft_consensus.cc:504] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.362025 16954 leader_election.cc:288] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 1 election: Requested vote from peers bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937), 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
I20260620 02:32:40.362576 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f890d08f474456c87facbe35d926df5" candidate_uuid: "e59be24c5e994c2988f086fe7eb2ed86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea31de2765b4b0ead4780ca30164deb"
I20260620 02:32:40.362625 16757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f890d08f474456c87facbe35d926df5" candidate_uuid: "e59be24c5e994c2988f086fe7eb2ed86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7640c157bb43818ad68268ebee8719"
I20260620 02:32:40.362813 16888 raft_consensus.cc:3037] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:40.362836 16757 raft_consensus.cc:3037] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:32:40.366375 16888 raft_consensus.cc:2454] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59be24c5e994c2988f086fe7eb2ed86 in term 1.
I20260620 02:32:40.366375 16757 raft_consensus.cc:2454] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59be24c5e994c2988f086fe7eb2ed86 in term 1.
I20260620 02:32:40.366896 16558 leader_election.cc:302] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7640c157bb43818ad68268ebee8719, e59be24c5e994c2988f086fe7eb2ed86; no voters:
I20260620 02:32:40.367282 16954 raft_consensus.cc:2781] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:32:40.367823 16954 raft_consensus.cc:686] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 LEADER]: Becoming Leader. State: Replica: e59be24c5e994c2988f086fe7eb2ed86, State: Running, Role: LEADER
I20260620 02:32:40.368337 16954 consensus_queue.cc:237] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:32:40.373447 16484 catalog_manager.cc:5495] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e59be24c5e994c2988f086fe7eb2ed86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } health_report { overall_health: UNKNOWN } } }
I20260620 02:32:40.422451 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.416871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.0.97.1:34655)
0620 02:32:40.417117 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.417121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.417144 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:32:40.417399 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.417403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.417484 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.417621 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:32:40.417945 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.418674 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.420283 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.420694 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.420798 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.421366 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:32:40.421877 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:32:40.421993 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.422161 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.422239 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:32:40.431839 16888 raft_consensus.cc:1261] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Refusing update from remote peer e59be24c5e994c2988f086fe7eb2ed86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:32:40.431839 16757 raft_consensus.cc:1261] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Refusing update from remote peer e59be24c5e994c2988f086fe7eb2ed86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:32:40.433260 16960 consensus_queue.cc:1035] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:32:40.433719 16954 consensus_queue.cc:1035] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:32:40.447202 16963 mvcc.cc:205] Tried to move back new op lower bound from 7298755626715934720 to 7298755626473009152. Current Snapshot: MvccSnapshot[applied={T|T < 7298755626715934720}]
I20260620 02:32:40.450309 16964 mvcc.cc:205] Tried to move back new op lower bound from 7298755626715934720 to 7298755626473009152. Current Snapshot: MvccSnapshot[applied={T|T < 7298755626715934720}]
I20260620 02:32:40.463852 16484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:32:40.464176 16484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:32:40.468412 16484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Sending DeleteTablet for 3 replicas of tablet 1f890d08f474456c87facbe35d926df5
I20260620 02:32:40.469987 16737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f890d08f474456c87facbe35d926df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32842
I20260620 02:32:40.470036 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f890d08f474456c87facbe35d926df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36658
I20260620 02:32:40.470993 16868 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f890d08f474456c87facbe35d926df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55546
I20260620 02:32:40.472350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16522
I20260620 02:32:40.479274 16974 tablet_replica.cc:330] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: stopping tablet replica
I20260620 02:32:40.480587 16974 raft_consensus.cc:2229] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:32:40.481889 16974 raft_consensus.cc:2258] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:32:40.486632 16974 ts_tablet_manager.cc:1905] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:32:40.494884 16974 ts_tablet_manager.cc:1918] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:32:40.497370 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.473911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.0.97.1:34655)
0620 02:32:40.474156 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.474164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.474187 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:32:40.474268 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.474275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.474380 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.474504 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:40.477261 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.478925 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.481459 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.482287 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.482479 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.487550 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:40.487568 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:40.487586 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:40.487645 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:40.492137 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.493217 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.493229 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.493233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.493406 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.493911 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.493916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.493920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.494406 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:40.494729 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.496328 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.496719 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:32:40.498832 16665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:40.499533 16972 tablet_replica.cc:330] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: stopping tablet replica
I20260620 02:32:40.500272 16972 raft_consensus.cc:2229] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:32:40.501171 16972 raft_consensus.cc:2258] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:32:40.504128 16972 ts_tablet_manager.cc:1905] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:32:40.504680 16974 log.cc:1196] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/1f890d08f474456c87facbe35d926df5
I20260620 02:32:40.505458 16974 ts_tablet_manager.cc:1939] T 1f890d08f474456c87facbe35d926df5 P bea31de2765b4b0ead4780ca30164deb: Deleting consensus metadata
I20260620 02:32:40.507844 16472 catalog_manager.cc:4841] TS bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937): tablet 1f890d08f474456c87facbe35d926df5 (table test-table [id=f6dc1be1e9f4497b88351c2a8308c4ca]) successfully deleted
I20260620 02:32:40.511792 16972 ts_tablet_manager.cc:1918] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:32:40.512007 16972 log.cc:1196] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/1f890d08f474456c87facbe35d926df5
I20260620 02:32:40.512471 16972 ts_tablet_manager.cc:1939] T 1f890d08f474456c87facbe35d926df5 P 8d7640c157bb43818ad68268ebee8719: Deleting consensus metadata
I20260620 02:32:40.514355 16469 catalog_manager.cc:4841] TS 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077): tablet 1f890d08f474456c87facbe35d926df5 (table test-table [id=f6dc1be1e9f4497b88351c2a8308c4ca]) successfully deleted
I20260620 02:32:40.618425 16973 tablet_replica.cc:330] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: stopping tablet replica
I20260620 02:32:40.618829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16522
W20260620 02:32:40.638818 16471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34655 (error 108)
I20260620 02:32:40.639055 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16674
I20260620 02:32:40.640974 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.640427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37484)
0620 02:32:40.640643 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.640785 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 02:32:40.641286 16471 catalog_manager.cc:4584] TS e59be24c5e994c2988f086fe7eb2ed86 (127.0.97.1:34655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f890d08f474456c87facbe35d926df5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
I20260620 02:32:40.654541 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.639968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.0.97.2:42077)
0620 02:32:40.640189 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.640196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.640219 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:32:40.640294 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.640302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.640407 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.640541 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:40.642125 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.642938 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.644302 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.644717 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.644826 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.647808 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:40.647828 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:40.647846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:40.647894 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:40.651170 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.652200 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.652207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.652210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.652363 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.652743 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.652747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.652748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.653312 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:40.653547 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.653942 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.654133 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:32:40.655391 16797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:40.695238 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.694624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37498)
0620 02:32:40.694928 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.695057 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 02:32:40.696597 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.696142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37502)
0620 02:32:40.696349 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.696440 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 02:32:40.747897 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16674
I20260620 02:32:40.757300 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.756802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37514)
0620 02:32:40.757035 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.757125 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 02:32:40.758395 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.758059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37518)
0620 02:32:40.758213 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.758291 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 02:32:40.766225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16805
I20260620 02:32:40.779933 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.766963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.97.3:34937)
0620 02:32:40.767157 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:32:40.767163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:32:40.767209 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:32:40.767344 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:32:40.767348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:32:40.767416 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:32:40.767525 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:32:40.768909 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.769692 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.771054 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:32:40.771467 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:32:40.771601 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:32:40.774354 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:32:40.774370 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:32:40.774386 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:32:40.774423 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:32:40.777143 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.777847 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.777863 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.777865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.778012 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:32:40.778393 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:32:40.778399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:32:40.778402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:32:40.778744 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:32:40.779050 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:32:40.779362 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 02:32:40.779583 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:32:40.780617 16928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:40.839295 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.838766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37526)
0620 02:32:40.838976 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.839076 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:32:40.840652 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:32:40.840243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34655 (local address 127.0.0.1:37540)
0620 02:32:40.840463 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 02:32:40.840533 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 02:32:40.874737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16805
I20260620 02:32:40.892400 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 16452
I20260620 02:32:40.893616 16515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:32:41.037752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 16452
I20260620 02:33:01.062326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45875
--webserver_interface=127.0.97.62
--webserver_port=35685
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:01.423695 16980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:01.424013 16980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:01.424108 16980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:01.431520 16980 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:33:01.431643 16980 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:33:01.431684 16980 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:33:01.431731 16980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:01.431788 16980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:01.431859 16980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:01.431905 16980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:01.431962 16980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:01.442152 16980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:01.443455 16980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:01.444836 16980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:01.457593 16985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:01.457677 16986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:01.457923 16980 server_base.cc:1034] running on GCE node
W20260620 02:33:01.457700 16988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:01.459149 16980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:01.460561 16980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:01.461796 16980 hybrid_clock.cc:648] HybridClock initialized: now 1781922781461729 us; error 38 us; skew 500 ppm
I20260620 02:33:01.462105 16980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:01 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:01.469226 16980 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:33:01.470805 16980 webserver.cc:460] Webserver started at http://127.0.97.62:35685/ using document root <none> and password file <none>
I20260620 02:33:01.471553 16980 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:01.471673 16980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:01.477595 16980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:33:01.480778 16995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:01.482376 16980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:33:01.482571 16980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "61f8392a77a7401b905389746eb42a7e"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "90137b48f52bef734064b580c894ffc8"
server_key_iv: "3b7a9bd35320ceb6f1c7153deef463bd"
server_key_version: "encryptionkey@0"
I20260620 02:33:01.483606 16980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:01.518636 16980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:01.523732 16980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:01.524061 16980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:01.544101 16980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45875
I20260620 02:33:01.544152 17046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45875 every 8 connection(s)
I20260620 02:33:01.545743 16980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:01.553176 17047 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:33:01.553403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 16980
I20260620 02:33:01.554374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34655
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45875
--webserver_port=43575
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:33:01.556770 17047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap starting.
I20260620 02:33:01.579247 17047 log.cc:824] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:01.596220 17047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:33:01.596868 17047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Bootstrap complete.
I20260620 02:33:01.606595 17047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:33:01.607671 17047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8392a77a7401b905389746eb42a7e, State: Initialized, Role: FOLLOWER
I20260620 02:33:01.608323 17047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:33:01.608592 17047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:01.608741 17047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:01.608927 17047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:33:01.612896 17047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:33:01.613432 17047 leader_election.cc:302] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f8392a77a7401b905389746eb42a7e; no voters:
I20260620 02:33:01.614149 17047 leader_election.cc:288] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:33:01.614310 17051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:33:01.615543 17051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [term 3 LEADER]: Becoming Leader. State: Replica: 61f8392a77a7401b905389746eb42a7e, State: Running, Role: LEADER
I20260620 02:33:01.616179 17051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } }
I20260620 02:33:01.616415 17047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:01.618727 17053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f8392a77a7401b905389746eb42a7e. Latest consensus state: current_term: 3 leader_uuid: "61f8392a77a7401b905389746eb42a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } } }
I20260620 02:33:01.619131 17053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:01.619730 17052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "61f8392a77a7401b905389746eb42a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f8392a77a7401b905389746eb42a7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45875 } } }
I20260620 02:33:01.620012 17052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:01.630239 17057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:01.640565 17057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:01.642246 17057 catalog_manager.cc:1259] Loaded cluster ID: 4e69bc904c3340ab860474d6bf69f6d5
I20260620 02:33:01.642455 17057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:01.647193 17057 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:01.649060 17057 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Loaded TSK: 0
I20260620 02:33:01.649322 17057 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61f8392a77a7401b905389746eb42a7e: Loaded TSK: 1
F20260620 02:33:01.654534 17057 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f49a4c309ad google::LogMessage::Fail() at ??:0
@ 0x7f49a4c3388a google::LogMessage::SendToLog() at ??:0
@ 0x7f49a4c304de google::LogMessage::Flush() at ??:0
@ 0x7f49a4c31a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f49a89947d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f49a898373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f49ba6631c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f49ba662783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f49ba6ee2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f49ba6ee0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f49ba7385e9 std::function<>::operator()() at ??:0
@ 0x7f49ba666cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f49ba664c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f49ba6ec610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f49ba6ec439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f49a787c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f49a787f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f49a787f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f49a785aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f49aab346db start_thread at ??:0
@ 0x7f49a3aca61f clone at ??:0
W20260620 02:33:01.953958 17049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:01.954355 17049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:01.954528 17049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:01.962268 17049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:01.962441 17049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:01.962635 17049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:01.973456 17049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43575
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:01.974910 17049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:01.976305 17049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:01.990098 17094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:01.990072 17095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:01.990171 17097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:01.991328 17049 server_base.cc:1034] running on GCE node
I20260620 02:33:01.991974 17049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:01.993062 17049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:01.994325 17049 hybrid_clock.cc:648] HybridClock initialized: now 1781922781994267 us; error 65 us; skew 500 ppm
I20260620 02:33:01.994673 17049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:01 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:01.997974 17049 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:33:01.999504 17049 webserver.cc:460] Webserver started at http://127.0.97.1:43575/ using document root <none> and password file <none>
I20260620 02:33:02.000239 17049 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:02.000397 17049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:02.006240 17049 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:33:02.009395 17104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:02.010803 17049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:33:02.011000 17049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "e59be24c5e994c2988f086fe7eb2ed86"
format_stamp: "Formatted at 2026-06-20 02:32:15 on dist-test-slave-r706"
server_key: "f0b019b5b5789d8e7b1c18a7b1b3ffdc"
server_key_iv: "0c2b623a63069e5b708a0799e589ecbd"
server_key_version: "encryptionkey@0"
I20260620 02:33:02.012001 17049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:02.028941 17049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:02.033375 17049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:02.033710 17049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:02.034799 17049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:02.036692 17111 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 02:33:02.043044 17049 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 02:33:02.043275 17049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 02:33:02.043448 17049 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 02:33:02.046965 17049 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 02:33:02.047120 17049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260620 02:33:02.047725 17111 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Bootstrap starting.
I20260620 02:33:02.076983 17111 log.cc:824] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:02.090991 17111 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:33:02.091856 17111 tablet_bootstrap.cc:492] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Bootstrap complete.
I20260620 02:33:02.093446 17111 ts_tablet_manager.cc:1397] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Time spent bootstrapping tablet: real 0.046s user 0.030s sys 0.015s
I20260620 02:33:02.094133 17049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34655
I20260620 02:33:02.094208 17218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34655 every 8 connection(s)
I20260620 02:33:02.095717 17049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:02.101524 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.098836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:52873)
0620 02:33:02.099841 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.100129 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":422,"threads_started":1}
W20260620 02:33:02.102288 17219 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45875 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45875: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
I20260620 02:33:02.103237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17049
I20260620 02:33:02.103514 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.103003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:51389)
0620 02:33:02.103227 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.103336 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:33:02.104137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42077
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45875
--webserver_port=39289
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:33:02.102612 17111 raft_consensus.cc:348] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:33:02.104435 17111 raft_consensus.cc:729] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59be24c5e994c2988f086fe7eb2ed86, State: Initialized, Role: FOLLOWER
I20260620 02:33:02.104730 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.104236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:50997)
0620 02:33:02.104467 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.104570 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260620 02:33:02.105103 17219 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:33:02.105228 17111 consensus_queue.cc:260] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:33:02.106990 17111 ts_tablet_manager.cc:1428] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 02:33:02.107088 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.106649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:52221)
0620 02:33:02.106824 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.106910 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:33:02.108537 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.107622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:43577)
0620 02:33:02.108203 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.108331 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":313,"threads_started":1}
W20260620 02:33:02.450502 17224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:02.450857 17224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:02.451051 17224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:02.458316 17224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:02.458508 17224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:02.458719 17224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:02.468750 17224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39289
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:02.470180 17224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:02.471489 17224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:02.484745 17229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:02.484797 17232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:02.484822 17230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:02.486104 17224 server_base.cc:1034] running on GCE node
I20260620 02:33:02.486821 17224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:02.487807 17224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:02.489099 17224 hybrid_clock.cc:648] HybridClock initialized: now 1781922782489022 us; error 60 us; skew 500 ppm
I20260620 02:33:02.489435 17224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:02 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:02.497753 17224 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:33:02.499517 17224 webserver.cc:460] Webserver started at http://127.0.97.2:39289/ using document root <none> and password file <none>
I20260620 02:33:02.500233 17224 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:02.500349 17224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:02.506045 17224 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:33:02.509146 17239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:02.510638 17224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:33:02.510802 17224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "8d7640c157bb43818ad68268ebee8719"
format_stamp: "Formatted at 2026-06-20 02:32:16 on dist-test-slave-r706"
server_key: "d1d7c822960c382aaf46499e74f4887d"
server_key_iv: "47ec786077985440b34c995b083bce48"
server_key_version: "encryptionkey@0"
I20260620 02:33:02.511799 17224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:02.524470 17224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:02.529171 17224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:02.529420 17224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:02.530429 17224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:02.531759 17224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:02.531857 17224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:02.531940 17224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:02.531997 17224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:02.567948 17224 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42077
I20260620 02:33:02.568053 17351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42077 every 8 connection(s)
I20260620 02:33:02.569535 17224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:02.574126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17224
I20260620 02:33:02.575031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34937
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45875
--webserver_port=35017
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:33:02.575073 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.572689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:56697)
0620 02:33:02.573510 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.573781 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":376,"threads_started":1}
W20260620 02:33:02.575978 17352 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45875 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45875: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
I20260620 02:33:02.577104 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.576599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:52977)
0620 02:33:02.576800 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.576904 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 02:33:02.578167 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:02.577720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:56705)
0620 02:33:02.577895 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:02.578009 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:33:02.578428 17352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:33:02.935863 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:02.936177 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:02.936352 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:02.944013 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:02.944154 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:02.944346 17355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:02.956624 17355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35017
--tserver_master_addrs=127.0.97.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:02.958582 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:02.960188 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:02.975637 17361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:02.975721 17360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:02.975696 17363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:02.976680 17355 server_base.cc:1034] running on GCE node
I20260620 02:33:02.977423 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:02.978482 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:02.979725 17355 hybrid_clock.cc:648] HybridClock initialized: now 1781922782979679 us; error 46 us; skew 500 ppm
I20260620 02:33:02.980070 17355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:02 dist-test-slave-r706 krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:02.983299 17355 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:33:02.984843 17355 webserver.cc:460] Webserver started at http://127.0.97.3:35017/ using document root <none> and password file <none>
I20260620 02:33:02.985662 17355 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:02.985816 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:02.991212 17355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 02:33:02.994222 17370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:02.995654 17355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:33:02.995847 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "bea31de2765b4b0ead4780ca30164deb"
format_stamp: "Formatted at 2026-06-20 02:32:17 on dist-test-slave-r706"
server_key: "84cb08aeaacf552acc885477af42cd01"
server_key_iv: "5eecf940a81d88a1d00e97aa5b781a89"
server_key_version: "encryptionkey@0"
I20260620 02:33:02.996856 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:03.024334 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:03.029409 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:03.029721 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:03.030788 17355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:03.032161 17355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:03.032291 17355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:03.032430 17355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:03.032522 17355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:03.069164 17355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34937
I20260620 02:33:03.069259 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34937 every 8 connection(s)
I20260620 02:33:03.070801 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:03.073803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17355
I20260620 02:33:03.076545 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.074290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:39933)
0620 02:33:03.075230 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.075535 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":399,"threads_started":1}
W20260620 02:33:03.077368 17483 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45875 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45875: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
I20260620 02:33:03.078338 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.077961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:48643)
0620 02:33:03.078132 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.078217 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:33:03.079257 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.078890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:51845)
0620 02:33:03.079046 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.079137 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:33:03.079557 17483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:33:03.110321 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.109439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:57013)
0620 02:33:03.110015 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.110113 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
I20260620 02:33:03.579993 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.579065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:48145)
0620 02:33:03.579637 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.579754 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":263,"threads_started":1}
I20260620 02:33:03.757493 17489 raft_consensus.cc:482] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:03.757835 17489 raft_consensus.cc:504] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:33:03.759479 17489 leader_election.cc:288] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937), 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
Jun 20 02:33:03 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 20 02:33:03 dist-test-slave-r706 krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:33:03.787238 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.760320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48305 (local address 127.0.97.2:42077)
0620 02:33:03.762711 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:03.762721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:03.762761 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:33:03.765240 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:03.765251 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:03.765812 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:03.766977 (+ 1165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:03.769165 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.770922 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.773251 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.774341 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.774561 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:03.779245 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:03.779253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:03.779266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:03.779309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:03.783138 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:03.784046 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:03.784054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:03.784060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:03.784231 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:03.784725 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:03.784730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:03.784732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:03.785380 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:03.785592 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:03.786398 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 02:33:03.786820 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2192,"thread_start_us":1932,"threads_started":1}
I20260620 02:33:03.787276 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.759778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42077 (local address 127.0.97.1:48305)
0620 02:33:03.760439 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.760496 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:03.761306 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:03.766505 (+ 5199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:03.766518 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:03.767679 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:03.768467 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:03.768490 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.771211 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.771217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:03.772951 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:03.772969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.774595 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.774601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:03.774813 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:03.775626 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:03.775657 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:03.778943 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:03.783311 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:03.783317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:03.783321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:03.783866 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:03.784378 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:03.784383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:03.784386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:03.784590 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:03.785529 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:03.785538 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:03.785872 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 02:33:03.786171 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:33:03.786757 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":220,"spinlock_wait_cycles":2432,"thread_start_us":2693,"threads_started":1}
I20260620 02:33:03.787441 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.759778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34937 (local address 127.0.97.1:39291)
0620 02:33:03.762746 (+ 2968us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:03.762763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:03.762885 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:03.766505 (+ 3620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:03.766520 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:03.767661 (+ 1141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:03.768470 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:03.768490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.771015 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.771024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:03.772952 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:03.772969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.774924 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.774929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:03.775072 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:03.775780 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:03.775798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:03.778650 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:03.783186 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:03.783198 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:03.783218 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:03.783871 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:03.784370 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:03.784380 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:03.784383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:03.784606 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:03.785529 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:03.785538 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:03.785872 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 02:33:03.786167 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 02:33:03.786950 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2865,"mutex_wait_us":483,"spinlock_wait_cycles":54784,"thread_start_us":715,"threads_started":1}
I20260620 02:33:03.787629 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:03.760293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39291 (local address 127.0.97.3:34937)
0620 02:33:03.762286 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:03.762309 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:03.762881 (+ 572us) server_negotiation.cc:407] Connection header received
0620 02:33:03.765184 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:03.765203 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:03.765812 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:03.766259 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:03.768729 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.770691 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.773202 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:03.774341 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:03.774561 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:03.779059 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:03.779070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:03.779086 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:03.779139 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:03.782983 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:03.784050 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:03.784057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:03.784063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:03.784234 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:03.784742 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:03.784747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:03.784749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:03.785367 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:03.785624 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:03.786413 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 02:33:03.786784 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":1567,"threads_started":1}
W20260620 02:33:03.791337 17105 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
W20260620 02:33:03.791602 17108 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
I20260620 02:33:03.791805 17108 leader_election.cc:302] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59be24c5e994c2988f086fe7eb2ed86; no voters: 8d7640c157bb43818ad68268ebee8719, bea31de2765b4b0ead4780ca30164deb
I20260620 02:33:03.792243 17489 raft_consensus.cc:2726] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:33:04.081399 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:04.080398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:36441)
0620 02:33:04.080904 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:04.081125 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":232,"threads_started":1}
I20260620 02:33:04.111817 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:04.111348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:36395)
0620 02:33:04.111561 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:04.111662 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 02:33:04.582134 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:04.581229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:60711)
0620 02:33:04.581804 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:04.581938 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":307,"threads_started":1}
I20260620 02:33:05.083588 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:05.082812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:39469)
0620 02:33:05.083330 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:05.083420 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":303,"threads_started":1}
I20260620 02:33:05.113713 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:05.112855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:57661)
0620 02:33:05.113443 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:05.113532 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":268,"threads_started":1}
I20260620 02:33:05.583952 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:05.583201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:52687)
0620 02:33:05.583655 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:05.583754 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260620 02:33:05.910064 17499 raft_consensus.cc:482] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:05.910295 17499 raft_consensus.cc:504] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:33:05.910975 17499 leader_election.cc:288] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937), 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
W20260620 02:33:05.912096 17105 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
W20260620 02:33:05.912334 17108 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
I20260620 02:33:05.912480 17108 leader_election.cc:302] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59be24c5e994c2988f086fe7eb2ed86; no voters: 8d7640c157bb43818ad68268ebee8719, bea31de2765b4b0ead4780ca30164deb
I20260620 02:33:05.912762 17499 raft_consensus.cc:2726] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:33:06.085628 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:06.084677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:49151)
0620 02:33:06.085327 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:06.085429 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":328,"threads_started":1}
I20260620 02:33:06.116052 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:06.114886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:38963)
0620 02:33:06.115612 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:06.115765 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
I20260620 02:33:06.585999 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:06.585194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:33495)
0620 02:33:06.585714 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:06.585806 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260620 02:33:07.087386 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:07.086643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:56763)
0620 02:33:07.087099 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:07.087189 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":208,"threads_started":1}
I20260620 02:33:07.118207 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:07.117327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:40969)
0620 02:33:07.117879 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:07.117992 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":278,"threads_started":1}
I20260620 02:33:07.588039 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:07.587174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:47887)
0620 02:33:07.587726 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:07.587850 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
I20260620 02:33:08.089334 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:08.088487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:60169)
0620 02:33:08.089063 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:08.089166 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":284,"threads_started":1}
I20260620 02:33:08.119884 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:08.119165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:45891)
0620 02:33:08.119640 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:08.119721 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260620 02:33:08.590262 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:08.589210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:42023)
0620 02:33:08.589819 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:08.589957 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":273,"threads_started":1}
I20260620 02:33:08.971069 17509 raft_consensus.cc:482] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:08.971278 17509 raft_consensus.cc:504] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:33:08.971978 17509 leader_election.cc:288] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937), 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
W20260620 02:33:08.972905 17108 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
W20260620 02:33:08.973129 17105 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
I20260620 02:33:08.973265 17105 leader_election.cc:302] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59be24c5e994c2988f086fe7eb2ed86; no voters: 8d7640c157bb43818ad68268ebee8719, bea31de2765b4b0ead4780ca30164deb
I20260620 02:33:08.973551 17509 raft_consensus.cc:2726] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:33:09.091245 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:09.090322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:55733)
0620 02:33:09.090957 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:09.091060 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":346,"threads_started":1}
I20260620 02:33:09.121820 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:09.120889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:50163)
0620 02:33:09.121515 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:09.121614 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":389,"threads_started":1}
I20260620 02:33:09.592461 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:09.591518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:48283)
0620 02:33:09.592112 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:09.592208 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":281,"threads_started":1}
I20260620 02:33:10.093323 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:10.092267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:38165)
0620 02:33:10.092960 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:10.093107 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":284,"threads_started":1}
I20260620 02:33:10.123453 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:10.122752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:35731)
0620 02:33:10.123206 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:10.123285 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":246,"threads_started":1}
I20260620 02:33:10.594403 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:10.593640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:40049)
0620 02:33:10.594119 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:10.594223 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260620 02:33:11.095446 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:11.094502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:45515)
0620 02:33:11.095104 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:11.095209 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":322,"threads_started":1}
I20260620 02:33:11.125555 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:11.124602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:52235)
0620 02:33:11.125247 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:11.125345 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":325,"threads_started":1}
I20260620 02:33:11.596472 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:11.595448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:56521)
0620 02:33:11.596182 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:11.596284 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":333,"threads_started":1}
I20260620 02:33:12.097462 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:12.096569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:56839)
0620 02:33:12.097218 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:12.097303 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":370,"threads_started":1}
I20260620 02:33:12.127537 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:12.126747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:49297)
0620 02:33:12.127254 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:12.127356 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260620 02:33:12.598531 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:12.597654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:57993)
0620 02:33:12.598251 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:12.598343 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":361,"threads_started":1}
I20260620 02:33:13.099485 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:13.098511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:36077)
0620 02:33:13.099076 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:13.099228 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
I20260620 02:33:13.129617 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:13.128595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:54279)
0620 02:33:13.129320 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:13.129421 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":627,"threads_started":1}
I20260620 02:33:13.580859 17524 raft_consensus.cc:482] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:13.581144 17524 raft_consensus.cc:504] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59be24c5e994c2988f086fe7eb2ed86" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34655 } } peers { permanent_uuid: "bea31de2765b4b0ead4780ca30164deb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34937 } } peers { permanent_uuid: "8d7640c157bb43818ad68268ebee8719" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42077 } }
I20260620 02:33:13.581845 17524 leader_election.cc:288] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937), 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077)
W20260620 02:33:13.582937 17108 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bea31de2765b4b0ead4780ca30164deb (127.0.97.3:34937): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
W20260620 02:33:13.583149 17105 leader_election.cc:341] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7640c157bb43818ad68268ebee8719 (127.0.97.2:42077): Not found: Tablet not found: 1f890d08f474456c87facbe35d926df5
I20260620 02:33:13.583258 17105 leader_election.cc:302] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59be24c5e994c2988f086fe7eb2ed86; no voters: 8d7640c157bb43818ad68268ebee8719, bea31de2765b4b0ead4780ca30164deb
I20260620 02:33:13.583498 17524 raft_consensus.cc:2726] T 1f890d08f474456c87facbe35d926df5 P e59be24c5e994c2988f086fe7eb2ed86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:33:13.600605 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:13.599621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:49029)
0620 02:33:13.600334 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:13.600429 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":347,"threads_started":1}
I20260620 02:33:14.101308 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:14.100395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:48385)
0620 02:33:14.100995 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:14.101141 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":303,"threads_started":1}
I20260620 02:33:14.131273 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:14.130543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:35291)
0620 02:33:14.130999 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:14.131081 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260620 02:33:14.602505 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:14.601664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:38619)
0620 02:33:14.602224 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:14.602317 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
I20260620 02:33:15.103027 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:15.102349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:39975)
0620 02:33:15.102779 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:15.102862 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260620 02:33:15.133174 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:15.132361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:59881)
0620 02:33:15.132828 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:15.132909 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
I20260620 02:33:15.604553 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:15.603751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:36111)
0620 02:33:15.604287 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:15.604370 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":279,"threads_started":1}
I20260620 02:33:16.104717 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:16.103937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:54137)
0620 02:33:16.104395 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:16.104476 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260620 02:33:16.134923 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:16.134096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:52691)
0620 02:33:16.134644 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:16.134727 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
I20260620 02:33:16.606590 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:16.605706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:37649)
0620 02:33:16.606277 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:16.606369 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":356,"threads_started":1}
I20260620 02:33:17.106505 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:17.105725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:44905)
0620 02:33:17.106221 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:17.106310 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":237,"threads_started":1}
I20260620 02:33:17.136963 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:17.136147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:43743)
0620 02:33:17.136675 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:17.136759 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":251,"threads_started":1}
I20260620 02:33:17.608695 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:17.607847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.2:49653)
0620 02:33:17.608295 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:17.608406 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
I20260620 02:33:18.086326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17049
I20260620 02:33:18.106009 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:18.087540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.0.97.1:34655)
0620 02:33:18.088287 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:18.088299 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:18.088340 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:33:18.088427 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:18.088466 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:18.088776 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:18.088984 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:18.090530 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:18.091538 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:18.092803 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:18.093432 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:18.093538 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:18.100068 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:18.100082 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:18.100098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:18.100136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:18.103176 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:18.103877 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:18.103884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:18.103887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:18.104045 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:18.104373 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:18.104376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:18.104378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:18.104736 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:18.104971 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:18.105377 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 02:33:18.105681 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":375,"threads_started":1}
I20260620 02:33:18.107285 17214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:18.108759 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:18.107865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.3:46997)
0620 02:33:18.108452 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:18.108587 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":295,"threads_started":1}
I20260620 02:33:18.198850 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:18.197692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45875 (local address 127.0.97.1:43687)
0620 02:33:18.198298 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:18.198480 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":260,"threads_started":1}
I20260620 02:33:18.198915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17049
I20260620 02:33:18.218636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17224
I20260620 02:33:18.232859 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:18.219466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.0.97.2:42077)
0620 02:33:18.220010 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:18.220015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:18.220030 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:33:18.220086 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:18.220090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:18.220151 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:18.220275 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:18.221836 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:18.222671 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:18.223997 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:18.224433 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:18.224575 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:18.227257 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:18.227262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:18.227265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:18.227300 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:18.230070 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:18.230770 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:18.230777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:18.230780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:18.230919 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:18.231313 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:18.231320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:18.231328 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:18.231613 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:18.231892 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:18.232185 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 02:33:18.232436 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
I20260620 02:33:18.234118 17347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:18.320068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17224
I20260620 02:33:18.339856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17355
I20260620 02:33:18.354847 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:18.340768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.0.97.3:34937)
0620 02:33:18.341367 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:18.341372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:18.341388 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:33:18.341451 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:18.341455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:18.341527 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:18.341657 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:18.343071 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:18.343868 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:18.345635 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:18.346084 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:18.346261 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:18.349101 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:18.349107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:18.349110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:18.349150 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:18.351888 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:18.352602 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:18.352608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:18.352612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:18.352772 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:18.353332 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:18.353338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:18.353341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:18.353633 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:18.353912 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:18.354226 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 02:33:18.354459 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":303,"threads_started":1}
I20260620 02:33:18.355679 17478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:18.441498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17355
2026-06-20T02:33:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64876 ms)
[----------] 30 tests from SecurityITest (312890 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:18 dist-test-slave-r706 krb5kdc[17548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:18 dist-test-slave-r706 krb5kdc[17548](info): set up 2 sockets
Jun 20 02:33:18 dist-test-slave-r706 krb5kdc[17548](info): commencing operation
krb5kdc: starting...
W20260620 02:33:19.581447 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:19 dist-test-slave-r706 krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:33:19.868834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37979
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:20.212368 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:20.212695 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:20.212877 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:20.220364 17561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:20.220503 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:20.220606 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:20.220693 17561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:20.220764 17561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:20.231105 17561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:20.232559 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:20.234004 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:20.246945 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:20.247562 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:20.247941 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:20.249130 17561 server_base.cc:1034] running on GCE node
I20260620 02:33:20.250075 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:20.251650 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:20.252892 17561 hybrid_clock.cc:648] HybridClock initialized: now 1781922800252851 us; error 41 us; skew 500 ppm
I20260620 02:33:20.253324 17561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:20 dist-test-slave-r706 krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:20.256569 17561 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:33:20.258322 17561 webserver.cc:460] Webserver started at http://127.0.97.62:41259/ using document root <none> and password file <none>
I20260620 02:33:20.259034 17561 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:20.259184 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:20.259519 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:20.262691 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "9fb4afbdfdc848ba92ca63fade7d2708"
format_stamp: "Formatted at 2026-06-20 02:33:20 on dist-test-slave-r706"
server_key: "5f8ae99ada2cf4d59af89d37d574f7eb"
server_key_iv: "1be67312e2f0c6ada2b5ed87adc006e1"
server_key_version: "encryptionkey@0"
I20260620 02:33:20.263475 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "9fb4afbdfdc848ba92ca63fade7d2708"
format_stamp: "Formatted at 2026-06-20 02:33:20 on dist-test-slave-r706"
server_key: "5f8ae99ada2cf4d59af89d37d574f7eb"
server_key_iv: "1be67312e2f0c6ada2b5ed87adc006e1"
server_key_version: "encryptionkey@0"
I20260620 02:33:20.268873 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:33:20.272385 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:20.273792 17561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:33:20.273986 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "9fb4afbdfdc848ba92ca63fade7d2708"
format_stamp: "Formatted at 2026-06-20 02:33:20 on dist-test-slave-r706"
server_key: "5f8ae99ada2cf4d59af89d37d574f7eb"
server_key_iv: "1be67312e2f0c6ada2b5ed87adc006e1"
server_key_version: "encryptionkey@0"
I20260620 02:33:20.274199 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:20.302305 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:20.306866 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:20.307217 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:20.327471 17561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37979
I20260620 02:33:20.327523 17627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37979 every 8 connection(s)
I20260620 02:33:20.329069 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:20.333117 17628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:20.336781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17561
I20260620 02:33:20.337059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:20.337936 388 external_mini_cluster.cc:1428] Setting key 75a0c3b0f006deffb0d2b71dff5eddc1
I20260620 02:33:20.347723 17628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: Bootstrap starting.
I20260620 02:33:20.352007 17628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:20.353550 17628 log.cc:824] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:33:20 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:20.357466 17628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: No bootstrap required, opened a new log
I20260620 02:33:20.365427 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:20.340565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.97.62:37979)
0620 02:33:20.341424 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:20.341435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:20.341478 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:33:20.343236 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:20.343263 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:20.343727 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:20.344110 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:20.346105 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:20.347506 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:20.349463 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:20.350212 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:20.350373 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:20.355924 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:20.355937 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:20.355957 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:20.356008 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:20.360283 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:20.361230 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:20.361240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:20.361249 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:20.361485 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:20.361908 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:20.361913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:20.361917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:20.362677 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:20.362963 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:20.363434 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 02:33:20.363940 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":424,"threads_started":1}
I20260620 02:33:20.368026 17628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37979 } }
I20260620 02:33:20.368537 17628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:20.368642 17628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb4afbdfdc848ba92ca63fade7d2708, State: Initialized, Role: FOLLOWER
I20260620 02:33:20.369269 17628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37979 } }
I20260620 02:33:20.369522 17628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:20.369668 17628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:20.369845 17628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:20.372089 17628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37979 } }
I20260620 02:33:20.372599 17628 leader_election.cc:302] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fb4afbdfdc848ba92ca63fade7d2708; no voters:
I20260620 02:33:20.373397 17628 leader_election.cc:288] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:20.373580 17633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:20.374526 17633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [term 1 LEADER]: Becoming Leader. State: Replica: 9fb4afbdfdc848ba92ca63fade7d2708, State: Running, Role: LEADER
I20260620 02:33:20.375052 17633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37979 } }
I20260620 02:33:20.375700 17628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:20.377877 17635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb4afbdfdc848ba92ca63fade7d2708. Latest consensus state: current_term: 1 leader_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37979 } } }
I20260620 02:33:20.378285 17635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:20.378937 17634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb4afbdfdc848ba92ca63fade7d2708" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37979 } } }
I20260620 02:33:20.379396 17634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708 [sys.catalog]: This master's current role is: LEADER
W20260620 02:33:20.389329 17649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:33:20.389487 17649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:33:20.389647 17639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:20.391184 17639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:20.399926 17639 catalog_manager.cc:1347] Generated new cluster ID: 364b90373d9a40ca9319239dacce3925
I20260620 02:33:20.400065 17639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:20.430895 17639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:20.432868 17639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:20.440475 17639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: Generated new TSK 0
I20260620 02:33:20.441458 17639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:33:20.524492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37979
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:20.870229 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:20.870584 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:20.870759 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:20.878541 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:20.878660 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:20.878822 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:20.889950 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:20.891390 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:20.892696 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:20.906500 17660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:20.906715 17659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:20.906736 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:20.907699 17654 server_base.cc:1034] running on GCE node
I20260620 02:33:20.908420 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:20.909504 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:20.910770 17654 hybrid_clock.cc:648] HybridClock initialized: now 1781922800910712 us; error 48 us; skew 500 ppm
I20260620 02:33:20.911154 17654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:20 dist-test-slave-r706 krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:20.914572 17654 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:33:20.916074 17654 webserver.cc:460] Webserver started at http://127.0.97.1:43837/ using document root <none> and password file <none>
I20260620 02:33:20.916828 17654 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:20.917038 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:20.917426 17654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:20.920431 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "159d1e7851344c868d14c1c27700a301"
format_stamp: "Formatted at 2026-06-20 02:33:20 on dist-test-slave-r706"
server_key: "6bcb7a79b564372c2d53c51833a66516"
server_key_iv: "c63de4f5c437d9c101d0c48231778ac2"
server_key_version: "encryptionkey@0"
I20260620 02:33:20.921337 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "159d1e7851344c868d14c1c27700a301"
format_stamp: "Formatted at 2026-06-20 02:33:20 on dist-test-slave-r706"
server_key: "6bcb7a79b564372c2d53c51833a66516"
server_key_iv: "c63de4f5c437d9c101d0c48231778ac2"
server_key_version: "encryptionkey@0"
I20260620 02:33:20.926664 17654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:33:20.930476 17669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:20.932065 17654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:20.932271 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "159d1e7851344c868d14c1c27700a301"
format_stamp: "Formatted at 2026-06-20 02:33:20 on dist-test-slave-r706"
server_key: "6bcb7a79b564372c2d53c51833a66516"
server_key_iv: "c63de4f5c437d9c101d0c48231778ac2"
server_key_version: "encryptionkey@0"
I20260620 02:33:20.932512 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:20.963698 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:20.968510 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:20.968855 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:20.970010 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:20.971458 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:20.971567 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:20.971721 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:20.971800 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:21.006477 17654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37331
I20260620 02:33:21.006541 17781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37331 every 8 connection(s)
I20260620 02:33:21.007818 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:21.013399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17654
I20260620 02:33:21.013628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:33:21.014473 388 external_mini_cluster.cc:1428] Setting key 41e150539f4e1d060779ef32198c4f3c
Jun 20 02:33:21 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:21.032099 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:21.010540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56209 (local address 127.0.97.62:37979)
0620 02:33:21.010992 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:21.010997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:21.012016 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 02:33:21.014661 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:21.014666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:21.014755 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:21.014858 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:21.016741 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.017596 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.020204 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.020636 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.020747 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:21.025389 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:21.025396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:21.025399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:21.025438 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:21.028389 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:21.029401 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:21.029409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:21.029414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:21.029562 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:21.030081 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:21.030087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:21.030089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:21.030392 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:21.030573 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:21.031401 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 02:33:21.031658 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260620 02:33:21.033243 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:21.011129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37979 (local address 127.0.97.1:56209)
0620 02:33:21.011832 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:21.011882 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:21.014248 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:21.015161 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:21.015170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:21.015816 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:21.016537 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:21.016584 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.017748 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.017757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:21.020009 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:21.020028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.020890 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.020900 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:21.021276 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:21.022046 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:21.022081 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:21.025103 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:21.028596 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:21.028607 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:21.028630 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:21.029249 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:21.029694 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:21.029702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:21.029706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:21.029931 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:21.030543 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:21.030553 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:21.030950 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 02:33:21.031262 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 02:33:21.031861 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":277,"threads_started":1}
I20260620 02:33:21.035106 17782 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37979
I20260620 02:33:21.035473 17782 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:21.036422 17782 heartbeater.cc:511] Master 127.0.97.62:37979 requested a full tablet report, sending...
I20260620 02:33:21.039026 17593 ts_manager.cc:194] Registered new tserver with Master: 159d1e7851344c868d14c1c27700a301 (127.0.97.1:37331)
I20260620 02:33:21.041411 17593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56209
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:33:21.080355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37979
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:21.422765 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:21.423126 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:21.423316 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:21.430864 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:21.431049 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:21.431226 17788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:21.442061 17788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:21.443480 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:21.444844 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:21.459668 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:21.459719 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:21.459774 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:21.461133 17788 server_base.cc:1034] running on GCE node
I20260620 02:33:21.461813 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:21.462992 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:21.464278 17788 hybrid_clock.cc:648] HybridClock initialized: now 1781922801464248 us; error 55 us; skew 500 ppm
I20260620 02:33:21.464654 17788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:21 dist-test-slave-r706 krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:21.468196 17788 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:33:21.469884 17788 webserver.cc:460] Webserver started at http://127.0.97.2:33705/ using document root <none> and password file <none>
I20260620 02:33:21.470621 17788 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:21.470780 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:21.471163 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:21.474269 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "108be1c67fad41fd9ffe34867d458328"
format_stamp: "Formatted at 2026-06-20 02:33:21 on dist-test-slave-r706"
server_key: "5fc1bcabaf85a51c943bfb1fffecb13e"
server_key_iv: "6cd62099d0793eddc6299f8c06e102bc"
server_key_version: "encryptionkey@0"
I20260620 02:33:21.475099 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "108be1c67fad41fd9ffe34867d458328"
format_stamp: "Formatted at 2026-06-20 02:33:21 on dist-test-slave-r706"
server_key: "5fc1bcabaf85a51c943bfb1fffecb13e"
server_key_iv: "6cd62099d0793eddc6299f8c06e102bc"
server_key_version: "encryptionkey@0"
I20260620 02:33:21.480873 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:33:21.484756 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:21.486239 17788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:33:21.486430 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "108be1c67fad41fd9ffe34867d458328"
format_stamp: "Formatted at 2026-06-20 02:33:21 on dist-test-slave-r706"
server_key: "5fc1bcabaf85a51c943bfb1fffecb13e"
server_key_iv: "6cd62099d0793eddc6299f8c06e102bc"
server_key_version: "encryptionkey@0"
I20260620 02:33:21.486642 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:21.508936 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:21.513597 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:21.513967 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:21.515029 17788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:21.516402 17788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:21.516515 17788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:21.516673 17788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:21.516748 17788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:21.552583 17788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33999
I20260620 02:33:21.552645 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33999 every 8 connection(s)
I20260620 02:33:21.553979 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:21.559664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17788
I20260620 02:33:21.559868 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:33:21.560417 388 external_mini_cluster.cc:1428] Setting key 75eb968185af8f36be11d135d5c69b14
Jun 20 02:33:21 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:21.578763 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:21.556515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57489 (local address 127.0.97.62:37979)
0620 02:33:21.557051 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:21.557057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:21.558228 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 02:33:21.560666 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:21.560671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:21.560746 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:21.560849 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:21.562861 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.563605 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.566280 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.566732 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.566834 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:21.571555 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:21.571560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:21.571564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:21.571599 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:21.574805 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:21.575775 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:21.575784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:21.575790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:21.575945 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:21.576443 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:21.576450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:21.576456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:21.576900 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:21.577115 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:21.577955 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 02:33:21.578242 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":276,"threads_started":1}
I20260620 02:33:21.579521 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:21.556952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37979 (local address 127.0.97.2:57489)
0620 02:33:21.558016 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:21.558062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:21.560329 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:21.561172 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:21.561181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:21.561883 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:21.562649 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:21.562676 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.563783 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.563792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:21.566117 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:21.566136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:21.566977 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:21.566986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:21.567326 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:21.568115 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:21.568151 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:21.571281 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:21.575041 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:21.575049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:21.575064 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:21.575654 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:21.576103 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:21.576107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:21.576109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:21.576321 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:21.577070 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:21.577076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:21.577536 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 02:33:21.577790 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 02:33:21.578315 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":380,"threads_started":1}
I20260620 02:33:21.581280 17916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37979
I20260620 02:33:21.581609 17916 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:21.582430 17916 heartbeater.cc:511] Master 127.0.97.62:37979 requested a full tablet report, sending...
I20260620 02:33:21.584167 17593 ts_manager.cc:194] Registered new tserver with Master: 108be1c67fad41fd9ffe34867d458328 (127.0.97.2:33999)
I20260620 02:33:21.585119 17593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57489
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:33:21.625281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37979
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:21.966621 17922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:21.966928 17922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:21.967105 17922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:21.974586 17922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:21.974725 17922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:21.974957 17922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:21.985637 17922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:21.986999 17922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:21.988355 17922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:22.001955 17927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:22.001998 17930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:22.002038 17928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:22.002768 17922 server_base.cc:1034] running on GCE node
I20260620 02:33:22.003473 17922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:22.004458 17922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:22.005703 17922 hybrid_clock.cc:648] HybridClock initialized: now 1781922802005649 us; error 54 us; skew 500 ppm
I20260620 02:33:22.006074 17922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:22 dist-test-slave-r706 krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:22.010147 17922 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:33:22.011776 17922 webserver.cc:460] Webserver started at http://127.0.97.3:37129/ using document root <none> and password file <none>
I20260620 02:33:22.012579 17922 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:22.012727 17922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:22.013118 17922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:22.016121 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "f6c44f7bf5b84ec7937896b591982d04"
format_stamp: "Formatted at 2026-06-20 02:33:22 on dist-test-slave-r706"
server_key: "6199a89a18d1f31efc825cf6d72c4676"
server_key_iv: "f31e6afc0e683546307690169de05538"
server_key_version: "encryptionkey@0"
I20260620 02:33:22.016955 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f6c44f7bf5b84ec7937896b591982d04"
format_stamp: "Formatted at 2026-06-20 02:33:22 on dist-test-slave-r706"
server_key: "6199a89a18d1f31efc825cf6d72c4676"
server_key_iv: "f31e6afc0e683546307690169de05538"
server_key_version: "encryptionkey@0"
I20260620 02:33:22.022493 17922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:33:22.026209 17937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:22.027583 17922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:33:22.027753 17922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "f6c44f7bf5b84ec7937896b591982d04"
format_stamp: "Formatted at 2026-06-20 02:33:22 on dist-test-slave-r706"
server_key: "6199a89a18d1f31efc825cf6d72c4676"
server_key_iv: "f31e6afc0e683546307690169de05538"
server_key_version: "encryptionkey@0"
I20260620 02:33:22.027953 17922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:22.045361 17782 heartbeater.cc:503] Master 127.0.97.62:37979 was elected leader, sending a full tablet report...
I20260620 02:33:22.054587 17922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:22.059235 17922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:22.059568 17922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:22.060676 17922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:22.062171 17922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:22.062299 17922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:22.062455 17922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:22.062531 17922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:22.098898 17922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46463
I20260620 02:33:22.098960 18049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46463 every 8 connection(s)
I20260620 02:33:22.100263 17922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:22.104439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 17922
I20260620 02:33:22.104662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:33:22.105237 388 external_mini_cluster.cc:1428] Setting key 4bb382b032fbd934d6a876dcfd066c5c
Jun 20 02:33:22 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:22.122829 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.103454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52069 (local address 127.0.97.62:37979)
0620 02:33:22.103950 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.103954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.105049 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 02:33:22.107330 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.107336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.107444 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.107573 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:22.109436 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.110236 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.111771 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.112182 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.112291 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.115821 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:22.115825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:22.115829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:22.115862 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:22.119182 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.120215 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.120222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.120228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.120412 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.120915 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.120920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.120923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.121285 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:22.121449 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.122196 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.122455 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
I20260620 02:33:22.123858 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.103975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37979 (local address 127.0.97.3:52069)
0620 02:33:22.104831 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:22.104884 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:22.106975 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:22.107826 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:22.107835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:22.108572 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:22.109263 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.109284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.110375 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.110379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.111631 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.111643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.112396 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.112401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.112601 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.113360 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:22.113384 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:22.115596 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:22.119353 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:22.119361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:22.119375 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:22.120070 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:22.120544 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:22.120550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:22.120554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:22.120786 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:22.121427 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:22.121435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:22.121798 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 02:33:22.122078 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 02:33:22.122633 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":378,"threads_started":1}
I20260620 02:33:22.125535 18050 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37979
I20260620 02:33:22.125880 18050 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:22.126616 18050 heartbeater.cc:511] Master 127.0.97.62:37979 requested a full tablet report, sending...
I20260620 02:33:22.128150 17593 ts_manager.cc:194] Registered new tserver with Master: f6c44f7bf5b84ec7937896b591982d04 (127.0.97.3:46463)
I20260620 02:33:22.129076 17593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52069
I20260620 02:33:22.134290 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:33:22.152679 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.139006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.0.97.62:37979)
0620 02:33:22.139262 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.139270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.139581 (+ 311us) server_negotiation.cc:407] Connection header received
0620 02:33:22.139722 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.139726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.139825 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.139917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:22.141510 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.142237 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.143717 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.144122 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.144222 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.146905 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:22.146910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:22.146913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:22.146947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:22.149829 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.150582 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.150587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.150589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.150740 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.151096 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.151099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.151101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.151464 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:22.151650 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.152048 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.152274 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:33:22.157145 17593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:33:22.164270 17593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:33:22.190719 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.177496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.0.97.3:46463)
0620 02:33:22.178755 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.178762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.180612 (+ 1850us) server_negotiation.cc:407] Connection header received
0620 02:33:22.180956 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.180964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.181236 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.181413 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:22.182670 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.184280 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.187871 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.188912 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.189084 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.189384 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.190033 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.190286 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":915,"threads_started":1}
I20260620 02:33:22.190924 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.177728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.0.97.1:37331)
0620 02:33:22.179253 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.179260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.179945 (+ 685us) server_negotiation.cc:407] Connection header received
0620 02:33:22.180463 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.180474 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.180793 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.181029 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:22.182119 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.183888 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.187824 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.189368 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.189514 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.189760 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.190385 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.190568 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1340,"thread_start_us":1173,"threads_started":1}
I20260620 02:33:22.191040 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.177114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37331 (local address 127.0.0.1:59716)
0620 02:33:22.179766 (+ 2652us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:22.179812 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:22.180061 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:22.181356 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:22.181366 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:22.181403 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:22.181866 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.181884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.184303 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.184309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.187647 (+ 3338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.187663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.189947 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.189953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.190180 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.190216 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:33:22.190316 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:33:22.190547 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2428,"mutex_wait_us":284,"thread_start_us":3004,"threads_started":1}
I20260620 02:33:22.191345 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.181161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.0.97.2:33999)
0620 02:33:22.181977 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.181989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.182758 (+ 769us) server_negotiation.cc:407] Connection header received
0620 02:33:22.182977 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.182985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.183201 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.183418 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:22.183986 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.185513 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.187696 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.189369 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.189514 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.189711 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.190775 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.190952 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":577,"threads_started":1}
I20260620 02:33:22.191982 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.177395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46463 (local address 127.0.0.1:53430)
0620 02:33:22.180491 (+ 3096us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:22.180510 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:22.180738 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:22.181785 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:22.181792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:22.181831 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:22.182385 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.182405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.184484 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.184492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.187647 (+ 3155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.187663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.189533 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.189539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.189737 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.189793 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:33:22.191235 (+ 1442us) client_negotiation.cc:240] Negotiation successful
0620 02:33:22.191433 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2970,"mutex_wait_us":114,"thread_start_us":2000,"threads_started":1}
I20260620 02:33:22.192047 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.180620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33999 (local address 127.0.0.1:38702)
0620 02:33:22.182629 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:22.182647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:22.182781 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:22.183453 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:22.183458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:22.183483 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:22.183814 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.183824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.185734 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.185738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.187487 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.187504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.189927 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.189937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.190283 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.190324 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:33:22.190477 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 02:33:22.190596 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1826,"mutex_wait_us":170,"thread_start_us":1643,"threads_started":1}
I20260620 02:33:22.195793 17851 tablet_service.cc:1467] Processing CreateTablet for tablet 11038ab5b5594ed4b9a49eb26b15e430 (DEFAULT_TABLE table=test-table [id=18e9e47ac36c455d962f715d13c92ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:22.195812 17985 tablet_service.cc:1467] Processing CreateTablet for tablet 11038ab5b5594ed4b9a49eb26b15e430 (DEFAULT_TABLE table=test-table [id=18e9e47ac36c455d962f715d13c92ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:22.195798 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 11038ab5b5594ed4b9a49eb26b15e430 (DEFAULT_TABLE table=test-table [id=18e9e47ac36c455d962f715d13c92ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:22.197346 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11038ab5b5594ed4b9a49eb26b15e430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:22.197368 17985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11038ab5b5594ed4b9a49eb26b15e430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:22.197368 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11038ab5b5594ed4b9a49eb26b15e430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:22.211858 18069 tablet_bootstrap.cc:492] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Bootstrap starting.
I20260620 02:33:22.215474 18069 tablet_bootstrap.cc:654] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:22.216661 18069 log.cc:824] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:22.218035 18071 tablet_bootstrap.cc:492] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Bootstrap starting.
I20260620 02:33:22.219058 18070 tablet_bootstrap.cc:492] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Bootstrap starting.
I20260620 02:33:22.223131 18071 tablet_bootstrap.cc:654] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:22.223415 18070 tablet_bootstrap.cc:654] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:22.224318 18071 log.cc:824] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:22.224434 18069 tablet_bootstrap.cc:492] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: No bootstrap required, opened a new log
I20260620 02:33:22.224792 18069 ts_tablet_manager.cc:1397] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260620 02:33:22.224792 18070 log.cc:824] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:22.227515 18070 tablet_bootstrap.cc:492] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: No bootstrap required, opened a new log
I20260620 02:33:22.227770 18071 tablet_bootstrap.cc:492] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: No bootstrap required, opened a new log
I20260620 02:33:22.227828 18070 ts_tablet_manager.cc:1397] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:33:22.228067 18071 ts_tablet_manager.cc:1397] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 02:33:22.236785 18070 raft_consensus.cc:348] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.236850 18069 raft_consensus.cc:348] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.237360 18069 raft_consensus.cc:374] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:22.237365 18070 raft_consensus.cc:374] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:22.237468 18070 raft_consensus.cc:729] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108be1c67fad41fd9ffe34867d458328, State: Initialized, Role: FOLLOWER
I20260620 02:33:22.237504 18069 raft_consensus.cc:729] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159d1e7851344c868d14c1c27700a301, State: Initialized, Role: FOLLOWER
I20260620 02:33:22.238135 18070 consensus_queue.cc:260] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.238261 18069 consensus_queue.cc:260] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.239460 17916 heartbeater.cc:503] Master 127.0.97.62:37979 was elected leader, sending a full tablet report...
I20260620 02:33:22.239964 18070 ts_tablet_manager.cc:1428] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 02:33:22.241712 18071 raft_consensus.cc:348] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.242250 18071 raft_consensus.cc:374] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:22.242367 18071 raft_consensus.cc:729] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c44f7bf5b84ec7937896b591982d04, State: Initialized, Role: FOLLOWER
I20260620 02:33:22.243045 18071 consensus_queue.cc:260] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.244460 18050 heartbeater.cc:503] Master 127.0.97.62:37979 was elected leader, sending a full tablet report...
I20260620 02:33:22.245311 18071 ts_tablet_manager.cc:1428] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260620 02:33:22.247514 18069 ts_tablet_manager.cc:1428] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
W20260620 02:33:22.260468 17783 tablet.cc:2369] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:33:22.306119 17917 tablet.cc:2369] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:22.349815 18075 raft_consensus.cc:482] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:22.350204 18075 raft_consensus.cc:504] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.352205 18075 leader_election.cc:288] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159d1e7851344c868d14c1c27700a301 (127.0.97.1:37331), f6c44f7bf5b84ec7937896b591982d04 (127.0.97.3:46463)
W20260620 02:33:22.352197 18051 tablet.cc:2369] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:22.359210 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.353426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48921 (local address 127.0.97.1:37331)
0620 02:33:22.353656 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.353663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.353854 (+ 191us) server_negotiation.cc:407] Connection header received
0620 02:33:22.354107 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.354111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.354172 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.354278 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:22.354841 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.355574 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.357466 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.358256 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.358414 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.358532 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.358824 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.358954 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 02:33:22.359401 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.352551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37331 (local address 127.0.97.2:48921)
0620 02:33:22.353692 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:22.353746 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:22.353953 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:22.354308 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:22.354313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:22.354341 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:22.354691 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.354700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.355708 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.355715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.357316 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.357328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.358496 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.358502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.358655 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.358679 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:33:22.358798 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 02:33:22.358932 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":4,"thread_start_us":1203,"threads_started":1}
I20260620 02:33:22.360065 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.353091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46213 (local address 127.0.97.3:46463)
0620 02:33:22.353364 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.353371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.354188 (+ 817us) server_negotiation.cc:407] Connection header received
0620 02:33:22.354403 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.354407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.354465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.354567 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:22.355135 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.356038 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.358042 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.358872 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.359050 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.359185 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.359663 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.359814 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:33:22.360692 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.352611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46463 (local address 127.0.97.2:46213)
0620 02:33:22.354068 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:22.354084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:22.354259 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:22.354590 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:22.354594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:22.354622 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:22.354987 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.354995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.356225 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.356232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.357861 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:22.357875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.359132 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.359139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:22.359413 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.359463 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 02:33:22.360273 (+ 810us) client_negotiation.cc:240] Negotiation successful
0620 02:33:22.360413 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"mutex_wait_us":25,"thread_start_us":730,"threads_started":1}
I20260620 02:33:22.360785 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11038ab5b5594ed4b9a49eb26b15e430" candidate_uuid: "108be1c67fad41fd9ffe34867d458328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159d1e7851344c868d14c1c27700a301" is_pre_election: true
I20260620 02:33:22.361516 17737 raft_consensus.cc:2454] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 108be1c67fad41fd9ffe34867d458328 in term 0.
I20260620 02:33:22.361778 18005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11038ab5b5594ed4b9a49eb26b15e430" candidate_uuid: "108be1c67fad41fd9ffe34867d458328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c44f7bf5b84ec7937896b591982d04" is_pre_election: true
I20260620 02:33:22.362246 18005 raft_consensus.cc:2454] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 108be1c67fad41fd9ffe34867d458328 in term 0.
I20260620 02:33:22.362282 17806 leader_election.cc:302] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 108be1c67fad41fd9ffe34867d458328, 159d1e7851344c868d14c1c27700a301; no voters:
I20260620 02:33:22.362726 18075 raft_consensus.cc:2781] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:33:22.362890 18075 raft_consensus.cc:482] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:33:22.363034 18075 raft_consensus.cc:3037] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:22.365592 18075 raft_consensus.cc:504] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.366411 18075 leader_election.cc:288] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [CANDIDATE]: Term 1 election: Requested vote from peers 159d1e7851344c868d14c1c27700a301 (127.0.97.1:37331), f6c44f7bf5b84ec7937896b591982d04 (127.0.97.3:46463)
I20260620 02:33:22.366952 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11038ab5b5594ed4b9a49eb26b15e430" candidate_uuid: "108be1c67fad41fd9ffe34867d458328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159d1e7851344c868d14c1c27700a301"
I20260620 02:33:22.367022 18005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11038ab5b5594ed4b9a49eb26b15e430" candidate_uuid: "108be1c67fad41fd9ffe34867d458328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c44f7bf5b84ec7937896b591982d04"
I20260620 02:33:22.367208 17737 raft_consensus.cc:3037] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:22.367296 18005 raft_consensus.cc:3037] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:22.369699 17737 raft_consensus.cc:2454] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 108be1c67fad41fd9ffe34867d458328 in term 1.
I20260620 02:33:22.369833 18005 raft_consensus.cc:2454] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 108be1c67fad41fd9ffe34867d458328 in term 1.
I20260620 02:33:22.370234 17806 leader_election.cc:302] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 108be1c67fad41fd9ffe34867d458328, 159d1e7851344c868d14c1c27700a301; no voters:
I20260620 02:33:22.370635 18075 raft_consensus.cc:2781] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:22.371251 18075 raft_consensus.cc:686] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 1 LEADER]: Becoming Leader. State: Replica: 108be1c67fad41fd9ffe34867d458328, State: Running, Role: LEADER
I20260620 02:33:22.371788 18075 consensus_queue.cc:237] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } }
I20260620 02:33:22.377090 17593 catalog_manager.cc:5495] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 108be1c67fad41fd9ffe34867d458328 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "108be1c67fad41fd9ffe34867d458328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "108be1c67fad41fd9ffe34867d458328" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33999 } health_report { overall_health: HEALTHY } } }
I20260620 02:33:22.434816 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.429305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.0.97.2:33999)
0620 02:33:22.429540 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.429545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.429561 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:33:22.429650 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.429654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.429745 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.429909 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:33:22.430327 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.431049 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.432518 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.432970 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.433138 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.433652 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:33:22.434181 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:33:22.434346 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.434426 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.434510 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:33:22.443800 17737 raft_consensus.cc:1261] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 1 FOLLOWER]: Refusing update from remote peer 108be1c67fad41fd9ffe34867d458328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:22.443800 18005 raft_consensus.cc:1261] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 1 FOLLOWER]: Refusing update from remote peer 108be1c67fad41fd9ffe34867d458328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:22.445138 18080 consensus_queue.cc:1035] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159d1e7851344c868d14c1c27700a301" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:22.445545 18075 consensus_queue.cc:1035] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c44f7bf5b84ec7937896b591982d04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:22.461767 18085 mvcc.cc:205] Tried to move back new op lower bound from 7298755798797709312 to 7298755798519640064. Current Snapshot: MvccSnapshot[applied={T|T < 7298755798797709312}]
I20260620 02:33:22.474419 17591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:22.474785 17591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:22.479259 17591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb4afbdfdc848ba92ca63fade7d2708: Sending DeleteTablet for 3 replicas of tablet 11038ab5b5594ed4b9a49eb26b15e430
I20260620 02:33:22.480707 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet 11038ab5b5594ed4b9a49eb26b15e430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59716
I20260620 02:33:22.481040 17985 tablet_service.cc:1514] Processing DeleteTablet for tablet 11038ab5b5594ed4b9a49eb26b15e430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53430
I20260620 02:33:22.481097 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet 11038ab5b5594ed4b9a49eb26b15e430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38702
I20260620 02:33:22.482987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17654
I20260620 02:33:22.491959 18094 tablet_replica.cc:330] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: stopping tablet replica
I20260620 02:33:22.492605 18094 raft_consensus.cc:2229] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:33:22.493217 18092 tablet_replica.cc:330] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: stopping tablet replica
I20260620 02:33:22.493527 18094 raft_consensus.cc:2258] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:22.493814 18092 raft_consensus.cc:2229] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:22.494381 18092 raft_consensus.cc:2258] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:22.496701 18092 ts_tablet_manager.cc:1905] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:22.499487 18093 tablet_replica.cc:330] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: stopping tablet replica
I20260620 02:33:22.500017 18093 raft_consensus.cc:2229] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:22.500512 18093 raft_consensus.cc:2258] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:22.502687 18093 ts_tablet_manager.cc:1905] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:22.504113 18092 ts_tablet_manager.cc:1918] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:22.504359 18092 log.cc:1196] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/11038ab5b5594ed4b9a49eb26b15e430
I20260620 02:33:22.505060 18092 ts_tablet_manager.cc:1939] T 11038ab5b5594ed4b9a49eb26b15e430 P 159d1e7851344c868d14c1c27700a301: Deleting consensus metadata
I20260620 02:33:22.507197 17579 catalog_manager.cc:4841] TS 159d1e7851344c868d14c1c27700a301 (127.0.97.1:37331): tablet 11038ab5b5594ed4b9a49eb26b15e430 (table test-table [id=18e9e47ac36c455d962f715d13c92ff1]) successfully deleted
Jun 20 02:33:22 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:33:22.510617 18093 ts_tablet_manager.cc:1918] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:22.510803 18093 log.cc:1196] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/11038ab5b5594ed4b9a49eb26b15e430
I20260620 02:33:22.511394 18093 ts_tablet_manager.cc:1939] T 11038ab5b5594ed4b9a49eb26b15e430 P f6c44f7bf5b84ec7937896b591982d04: Deleting consensus metadata
I20260620 02:33:22.513753 17580 catalog_manager.cc:4841] TS f6c44f7bf5b84ec7937896b591982d04 (127.0.97.3:46463): tablet 11038ab5b5594ed4b9a49eb26b15e430 (table test-table [id=18e9e47ac36c455d962f715d13c92ff1]) successfully deleted
I20260620 02:33:22.514165 18094 ts_tablet_manager.cc:1905] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:22.518054 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.485026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.0.97.1:37331)
0620 02:33:22.485269 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.485278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.498843 (+ 13565us) server_negotiation.cc:407] Connection header received
0620 02:33:22.499065 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.499071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.499154 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.499260 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:22.501420 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.502612 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.504575 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.505293 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.505428 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.511323 (+ 5895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:22.511343 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:22.511366 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:22.511421 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:22.514913 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.515690 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.515697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.515700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.515836 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.516252 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.516264 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.516266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.516637 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:22.516971 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.517314 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.517566 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:33:22.518800 17777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:22.520368 18094 ts_tablet_manager.cc:1918] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:22.520638 18094 log.cc:1196] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/11038ab5b5594ed4b9a49eb26b15e430
I20260620 02:33:22.521687 18094 ts_tablet_manager.cc:1939] T 11038ab5b5594ed4b9a49eb26b15e430 P 108be1c67fad41fd9ffe34867d458328: Deleting consensus metadata
I20260620 02:33:22.524040 17579 catalog_manager.cc:4841] TS 108be1c67fad41fd9ffe34867d458328 (127.0.97.2:33999): tablet 11038ab5b5594ed4b9a49eb26b15e430 (table test-table [id=18e9e47ac36c455d962f715d13c92ff1]) successfully deleted
I20260620 02:33:22.611480 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17654
I20260620 02:33:22.633484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17788
Jun 20 02:33:22 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:33:22.651728 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.634337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.97.2:33999)
0620 02:33:22.634550 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.634556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.634580 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:33:22.634649 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.634654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.634750 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.634879 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:22.636175 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.637489 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.638856 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.639560 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.639688 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.644480 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:22.644503 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:22.644525 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:22.644593 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:22.648690 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.649522 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.649533 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.649538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.649714 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.650064 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.650069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.650073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.650550 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:22.650922 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.651091 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.651333 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:33:22.652552 17911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:22.781468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17788
I20260620 02:33:22.800829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17922
Jun 20 02:33:22 dist-test-slave-r706 krb5kdc[17548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:33:22.815750 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:22.801660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.0.97.3:46463)
0620 02:33:22.801837 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:22.801841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:22.801863 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:33:22.801987 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:22.801991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:22.802057 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:22.802151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:22.803428 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.804139 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.805483 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:22.805896 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:22.805995 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:22.810337 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:22.810352 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:22.810369 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:22.810408 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:22.813331 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.813988 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.813994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.813997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.814135 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:22.814498 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:22.814501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:22.814503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:22.814840 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:22.815053 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:22.815324 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 02:33:22.815487 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:33:22.816398 18045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:22.905917 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17922
I20260620 02:33:22.923549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 17561
I20260620 02:33:22.924752 17623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:23.039918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 17561
2026-06-20T02:33:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4605 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T02:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:23Z Disabled control of system clock
I20260620 02:33:23.130920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37689
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:33:23.475183 18103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:23.475631 18103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:23.475785 18103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:23.483333 18103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:23.483469 18103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:23.483508 18103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:23.483558 18103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:23.483632 18103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:23.493203 18103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:23.494578 18103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:23.496397 18103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:23.509136 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:23.509100 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:23.509047 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:23.510001 18103 server_base.cc:1034] running on GCE node
I20260620 02:33:23.510762 18103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:23.512133 18103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:23.513335 18103 hybrid_clock.cc:648] HybridClock initialized: now 1781922803513296 us; error 36 us; skew 500 ppm
I20260620 02:33:23.513715 18103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:23.520431 18103 webserver.cc:460] Webserver started at http://127.0.97.62:34435/ using document root <none> and password file <none>
I20260620 02:33:23.521302 18103 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:23.521432 18103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:23.521806 18103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:23.524981 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "7f18780ca6c3452194591885183ea102"
format_stamp: "Formatted at 2026-06-20 02:33:23 on dist-test-slave-r706"
server_key: "a8322de0558ccd5a3249de7e0ad5e82c"
server_key_iv: "a588343907c8537e359ea734665ef1e8"
server_key_version: "encryptionkey@0"
I20260620 02:33:23.525923 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f18780ca6c3452194591885183ea102"
format_stamp: "Formatted at 2026-06-20 02:33:23 on dist-test-slave-r706"
server_key: "a8322de0558ccd5a3249de7e0ad5e82c"
server_key_iv: "a588343907c8537e359ea734665ef1e8"
server_key_version: "encryptionkey@0"
I20260620 02:33:23.531316 18103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:33:23.534993 18117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:23.536396 18103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:33:23.536612 18103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "7f18780ca6c3452194591885183ea102"
format_stamp: "Formatted at 2026-06-20 02:33:23 on dist-test-slave-r706"
server_key: "a8322de0558ccd5a3249de7e0ad5e82c"
server_key_iv: "a588343907c8537e359ea734665ef1e8"
server_key_version: "encryptionkey@0"
I20260620 02:33:23.536859 18103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:23.569041 18103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:23.569972 18103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:23.570293 18103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:23.590265 18103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37689
I20260620 02:33:23.590284 18168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37689 every 8 connection(s)
I20260620 02:33:23.591776 18103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:23.596320 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:23.599329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18103
I20260620 02:33:23.599537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:23.600114 388 external_mini_cluster.cc:1428] Setting key 821807ca7fa6e7701863f45420ffc206
I20260620 02:33:23.610074 18169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102: Bootstrap starting.
I20260620 02:33:23.613515 18169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:23.614735 18169 log.cc:824] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:23.616480 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:23.602728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.0.97.62:37689)
0620 02:33:23.603910 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:23.603922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:23.603982 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:33:23.605991 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:23.606016 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:23.606551 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:23.607050 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:23.608278 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:23.609694 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:23.611349 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:23.612120 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:23.612350 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:23.613738 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:23.613750 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:23.613772 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:23.614371 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:23.614472 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:23.614495 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:23.614644 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:23.614882 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 02:33:23.615201 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":452,"threads_started":1}
I20260620 02:33:23.618271 18169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102: No bootstrap required, opened a new log
I20260620 02:33:23.627765 18169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f18780ca6c3452194591885183ea102" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37689 } }
I20260620 02:33:23.628218 18169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:23.628362 18169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f18780ca6c3452194591885183ea102, State: Initialized, Role: FOLLOWER
I20260620 02:33:23.628957 18169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f18780ca6c3452194591885183ea102" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37689 } }
I20260620 02:33:23.629312 18169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:23.629470 18169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:23.629662 18169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:23.632262 18169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f18780ca6c3452194591885183ea102" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37689 } }
I20260620 02:33:23.632920 18169 leader_election.cc:302] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f18780ca6c3452194591885183ea102; no voters:
I20260620 02:33:23.633638 18169 leader_election.cc:288] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:23.633945 18174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:23.634907 18174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [term 1 LEADER]: Becoming Leader. State: Replica: 7f18780ca6c3452194591885183ea102, State: Running, Role: LEADER
I20260620 02:33:23.635566 18174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f18780ca6c3452194591885183ea102" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37689 } }
I20260620 02:33:23.636368 18169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:23.638290 18176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f18780ca6c3452194591885183ea102. Latest consensus state: current_term: 1 leader_uuid: "7f18780ca6c3452194591885183ea102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f18780ca6c3452194591885183ea102" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37689 } } }
I20260620 02:33:23.638725 18176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:23.638737 18175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f18780ca6c3452194591885183ea102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f18780ca6c3452194591885183ea102" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37689 } } }
I20260620 02:33:23.639220 18175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:23.644827 18179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:23.650539 18179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:23.665122 18179 catalog_manager.cc:1347] Generated new cluster ID: 8eba809bf27447648a487a811f5d49d2
I20260620 02:33:23.665535 18179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:23.683846 18179 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:23.685238 18179 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:23.695209 18179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102: Generated new TSK 0
I20260620 02:33:23.696089 18179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:23.710750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37689
--builtin_ntp_servers=127.0.97.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:33:24.072579 18193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:24.072976 18193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:24.073205 18193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:24.081068 18193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:24.081230 18193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:24.081403 18193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:24.091066 18193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:24.092466 18193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:24.094648 18193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:24.110767 18199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:24.110744 18198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:24.110800 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:24.111744 18193 server_base.cc:1034] running on GCE node
I20260620 02:33:24.112504 18193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:24.113803 18193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:24.115192 18193 hybrid_clock.cc:648] HybridClock initialized: now 1781922804115116 us; error 67 us; skew 500 ppm
I20260620 02:33:24.115588 18193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:24.118533 18193 webserver.cc:460] Webserver started at http://127.0.97.1:41131/ using document root <none> and password file <none>
I20260620 02:33:24.119292 18193 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:24.119434 18193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:24.119781 18193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:24.122861 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "61fc9723955d47079fb1e90d1658f4cd"
format_stamp: "Formatted at 2026-06-20 02:33:24 on dist-test-slave-r706"
server_key: "77610608d90f082c79bc5d190fed7876"
server_key_iv: "73f2014124315805c3c361f6c9b4cbd5"
server_key_version: "encryptionkey@0"
I20260620 02:33:24.123692 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "61fc9723955d47079fb1e90d1658f4cd"
format_stamp: "Formatted at 2026-06-20 02:33:24 on dist-test-slave-r706"
server_key: "77610608d90f082c79bc5d190fed7876"
server_key_iv: "73f2014124315805c3c361f6c9b4cbd5"
server_key_version: "encryptionkey@0"
I20260620 02:33:24.131191 18193 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 02:33:24.135432 18207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:24.136943 18193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:33:24.137233 18193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "61fc9723955d47079fb1e90d1658f4cd"
format_stamp: "Formatted at 2026-06-20 02:33:24 on dist-test-slave-r706"
server_key: "77610608d90f082c79bc5d190fed7876"
server_key_iv: "73f2014124315805c3c361f6c9b4cbd5"
server_key_version: "encryptionkey@0"
I20260620 02:33:24.137483 18193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:24.151158 18193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:24.152398 18193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:24.152726 18193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:24.153983 18193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:24.155650 18193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:24.155786 18193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:24.155942 18193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:24.156035 18193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:24.190374 18193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40593
I20260620 02:33:24.190429 18319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40593 every 8 connection(s)
I20260620 02:33:24.191740 18193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:24.192472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18193
I20260620 02:33:24.192692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:33:24.193310 388 external_mini_cluster.cc:1428] Setting key 5d4b2c22f32522065396773325c7525c
I20260620 02:33:24.196504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37689
--builtin_ntp_servers=127.0.97.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:33:24.205756 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:24.194707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59117 (local address 127.0.97.62:37689)
0620 02:33:24.195167 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:24.195173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:24.196709 (+ 1536us) server_negotiation.cc:407] Connection header received
0620 02:33:24.198993 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:24.198998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:24.199092 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:24.199224 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:24.200329 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.201136 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.202733 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.203161 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.203303 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:24.204765 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:24.204771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:24.204776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:24.204806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:24.204845 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:24.204857 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:24.204958 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:24.205258 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 02:33:24.205366 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260620 02:33:24.206749 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:24.195463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37689 (local address 127.0.97.1:59117)
0620 02:33:24.196532 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:24.196599 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:24.198701 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:24.199471 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:24.199480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:24.199571 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:24.200148 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:24.200174 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.201304 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.201308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:24.202574 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:24.202586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.203417 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.203425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:24.203667 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:24.204430 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:24.204433 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:24.204474 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:24.204554 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:24.204959 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:24.204973 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:24.204983 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:33:24.205231 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 02:33:24.205635 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":347,"threads_started":1}
I20260620 02:33:24.208503 18320 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37689
I20260620 02:33:24.208920 18320 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:24.209724 18320 heartbeater.cc:511] Master 127.0.97.62:37689 requested a full tablet report, sending...
I20260620 02:33:24.212229 18134 ts_manager.cc:194] Registered new tserver with Master: 61fc9723955d47079fb1e90d1658f4cd (127.0.97.1:40593)
I20260620 02:33:24.214538 18134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59117
W20260620 02:33:24.545511 18324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:24.545814 18324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:24.545961 18324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:24.553680 18324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:24.553808 18324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:24.553923 18324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:24.563345 18324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:24.564668 18324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:24.566613 18324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:24.580857 18329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:24.580826 18330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:24.580911 18332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:24.581779 18324 server_base.cc:1034] running on GCE node
I20260620 02:33:24.582420 18324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:24.583436 18324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:24.584666 18324 hybrid_clock.cc:648] HybridClock initialized: now 1781922804584612 us; error 37 us; skew 500 ppm
I20260620 02:33:24.585079 18324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:24.587754 18324 webserver.cc:460] Webserver started at http://127.0.97.2:33561/ using document root <none> and password file <none>
I20260620 02:33:24.588552 18324 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:24.588682 18324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:24.589083 18324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:24.592183 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "e822525958d842be9b74babd12539d44"
format_stamp: "Formatted at 2026-06-20 02:33:24 on dist-test-slave-r706"
server_key: "2fe847128415f77d308b9f53255d8a1b"
server_key_iv: "49bb490db7e9739fd13dae149cf31325"
server_key_version: "encryptionkey@0"
I20260620 02:33:24.593176 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e822525958d842be9b74babd12539d44"
format_stamp: "Formatted at 2026-06-20 02:33:24 on dist-test-slave-r706"
server_key: "2fe847128415f77d308b9f53255d8a1b"
server_key_iv: "49bb490db7e9739fd13dae149cf31325"
server_key_version: "encryptionkey@0"
I20260620 02:33:24.599035 18324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:33:24.603120 18338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:24.604732 18324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:33:24.604885 18324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "e822525958d842be9b74babd12539d44"
format_stamp: "Formatted at 2026-06-20 02:33:24 on dist-test-slave-r706"
server_key: "2fe847128415f77d308b9f53255d8a1b"
server_key_iv: "49bb490db7e9739fd13dae149cf31325"
server_key_version: "encryptionkey@0"
I20260620 02:33:24.605154 18324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:24.625759 18324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:24.626647 18324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:24.626959 18324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:24.628070 18324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:24.629604 18324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:24.629701 18324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:24.629784 18324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:24.629840 18324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:24.667946 18324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43793
I20260620 02:33:24.668053 18450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43793 every 8 connection(s)
I20260620 02:33:24.669574 18324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:24.678066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18324
I20260620 02:33:24.678247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:33:24.678769 388 external_mini_cluster.cc:1428] Setting key 05c26d38ae3fdd571aa1b5790f77a031
I20260620 02:33:24.682117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37689
--builtin_ntp_servers=127.0.97.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:33:24.685637 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:24.673282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38571 (local address 127.0.97.62:37689)
0620 02:33:24.673522 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:24.673529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:24.675132 (+ 1603us) server_negotiation.cc:407] Connection header received
0620 02:33:24.677130 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:24.677135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:24.677234 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:24.677339 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:24.678757 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.679836 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.682357 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.682834 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.682949 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:24.684824 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:24.684829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:24.684833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:24.684854 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:24.684889 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:24.684901 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:24.684975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:24.685311 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 02:33:24.685406 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:33:24.687275 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:24.673836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37689 (local address 127.0.97.2:38571)
0620 02:33:24.674911 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:24.674967 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:24.676696 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:24.677589 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:24.677598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:24.677723 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:24.678527 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:24.678551 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.680052 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.680057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:24.682106 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:24.682128 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:24.683103 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:24.683114 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:24.683498 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:24.684375 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:24.684381 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:24.684442 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:24.684575 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:24.685043 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:24.685056 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:24.685067 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:33:24.685281 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 02:33:24.685760 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":364,"threads_started":1}
I20260620 02:33:24.689491 18451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37689
I20260620 02:33:24.689859 18451 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:24.690831 18451 heartbeater.cc:511] Master 127.0.97.62:37689 requested a full tablet report, sending...
I20260620 02:33:24.692709 18134 ts_manager.cc:194] Registered new tserver with Master: e822525958d842be9b74babd12539d44 (127.0.97.2:43793)
I20260620 02:33:24.693722 18134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38571
W20260620 02:33:25.048105 18454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:25.048466 18454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:25.048640 18454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:25.056653 18454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:25.056839 18454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:25.057102 18454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:25.067070 18454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:25.068480 18454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:25.070502 18454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:25.086520 18459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:25.086833 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:25.086833 18462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:25.087671 18454 server_base.cc:1034] running on GCE node
I20260620 02:33:25.088378 18454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:25.089704 18454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:25.090945 18454 hybrid_clock.cc:648] HybridClock initialized: now 1781922805090871 us; error 70 us; skew 500 ppm
I20260620 02:33:25.091305 18454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:25.094089 18454 webserver.cc:460] Webserver started at http://127.0.97.3:33753/ using document root <none> and password file <none>
I20260620 02:33:25.094899 18454 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:25.095023 18454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:25.095387 18454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:25.098541 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "701230de4dba46dfa39a691e1314b5aa"
format_stamp: "Formatted at 2026-06-20 02:33:25 on dist-test-slave-r706"
server_key: "b35ddcce3b4c982288ec55ca3981eebd"
server_key_iv: "337a99767cf971d84a95062e1259af91"
server_key_version: "encryptionkey@0"
I20260620 02:33:25.099408 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "701230de4dba46dfa39a691e1314b5aa"
format_stamp: "Formatted at 2026-06-20 02:33:25 on dist-test-slave-r706"
server_key: "b35ddcce3b4c982288ec55ca3981eebd"
server_key_iv: "337a99767cf971d84a95062e1259af91"
server_key_version: "encryptionkey@0"
I20260620 02:33:25.106148 18454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:33:25.110466 18468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:25.112370 18454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:33:25.112599 18454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "701230de4dba46dfa39a691e1314b5aa"
format_stamp: "Formatted at 2026-06-20 02:33:25 on dist-test-slave-r706"
server_key: "b35ddcce3b4c982288ec55ca3981eebd"
server_key_iv: "337a99767cf971d84a95062e1259af91"
server_key_version: "encryptionkey@0"
I20260620 02:33:25.112926 18454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:25.149025 18454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:25.150306 18454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:25.150691 18454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:25.152136 18454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:25.154606 18454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:25.154754 18454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:25.154898 18454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:25.155001 18454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:25.196360 18454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37661
I20260620 02:33:25.196399 18580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37661 every 8 connection(s)
I20260620 02:33:25.198458 18454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:25.204427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18454
I20260620 02:33:25.204635 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:33:25.205236 388 external_mini_cluster.cc:1428] Setting key 9977f6e41166b208a2c67fe013abc497
I20260620 02:33:25.215682 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.201669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51781 (local address 127.0.97.62:37689)
0620 02:33:25.202136 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.202141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.203752 (+ 1611us) server_negotiation.cc:407] Connection header received
0620 02:33:25.206349 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.206355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.206463 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.206610 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:25.208343 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.209265 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.212235 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.212704 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.212836 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.214824 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:25.214829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:25.214832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:25.214854 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:25.214895 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:25.214907 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:25.214978 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.215277 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.215368 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":259,"threads_started":1}
I20260620 02:33:25.217487 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.202445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37689 (local address 127.0.97.3:51781)
0620 02:33:25.203555 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.203601 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:25.206032 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:25.206944 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:25.206961 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:25.207136 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:25.208130 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.208160 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.209426 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.209434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.212007 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.212031 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.213021 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.213032 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.213444 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.214351 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:25.214357 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:25.214435 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:25.214570 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:25.215006 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:25.215019 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:25.215030 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:33:25.215264 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 02:33:25.215787 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":363,"threads_started":1}
I20260620 02:33:25.218104 18320 heartbeater.cc:503] Master 127.0.97.62:37689 was elected leader, sending a full tablet report...
I20260620 02:33:25.219290 18581 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37689
I20260620 02:33:25.219623 18581 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:25.220633 18581 heartbeater.cc:511] Master 127.0.97.62:37689 requested a full tablet report, sending...
I20260620 02:33:25.222218 18133 ts_manager.cc:194] Registered new tserver with Master: 701230de4dba46dfa39a691e1314b5aa (127.0.97.3:37661)
I20260620 02:33:25.223093 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51781
I20260620 02:33:25.235100 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:33:25.246876 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.239819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.0.97.62:37689)
0620 02:33:25.240081 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.240091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.240631 (+ 540us) server_negotiation.cc:407] Connection header received
0620 02:33:25.240838 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.240842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.240955 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.241119 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:25.241712 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.242554 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.244043 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.244464 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.244613 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.245821 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:25.245827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:25.245832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:25.245897 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:25.245938 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:25.245959 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:25.246055 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.246355 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.246501 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:33:25.252326 18133 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:33:25.262809 18133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:33:25.289970 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.277869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37661 (local address 127.0.0.1:48104)
0620 02:33:25.280919 (+ 3050us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.280979 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:25.281300 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:25.282334 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:25.282342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:25.282381 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:25.282943 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.282961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.284835 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.284840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.287141 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.287154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.288726 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.288732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.288949 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.289045 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 02:33:25.289299 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 02:33:25.289535 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2862,"mutex_wait_us":28,"spinlock_wait_cycles":13056,"thread_start_us":1604,"threads_started":1}
I20260620 02:33:25.291394 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.278443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.0.97.3:37661)
0620 02:33:25.279312 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.279318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.281118 (+ 1800us) server_negotiation.cc:407] Connection header received
0620 02:33:25.281552 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.281561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.281821 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.281987 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:25.283170 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.284679 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.287345 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.288524 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.288678 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.288965 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.289357 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.290989 (+ 1632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":585,"threads_started":1}
I20260620 02:33:25.291458 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.279979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43793 (local address 127.0.0.1:56978)
0620 02:33:25.282912 (+ 2933us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.282932 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:25.283063 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:25.283730 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:25.283735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:25.283763 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:25.284156 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.284168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.285926 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.285935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.288309 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.288320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.290622 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.290637 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.290837 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.290874 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:33:25.290975 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:33:25.291115 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2780,"mutex_wait_us":21,"thread_start_us":601,"threads_started":1}
I20260620 02:33:25.291534 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.277207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.0.97.2:43793)
0620 02:33:25.280191 (+ 2984us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.280198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.283048 (+ 2850us) server_negotiation.cc:407] Connection header received
0620 02:33:25.283243 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.283253 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.283515 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.283740 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:25.284345 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.285739 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.288498 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.290159 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.290288 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.290500 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.291048 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.291221 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2815,"thread_start_us":2723,"threads_started":1}
I20260620 02:33:25.291527 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.277976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.0.97.1:40593)
0620 02:33:25.278792 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.278804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.281118 (+ 2314us) server_negotiation.cc:407] Connection header received
0620 02:33:25.281562 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.281569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.281821 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.281987 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:25.283586 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.284987 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.288346 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.289976 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.290121 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.290398 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.290959 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.291150 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":1528,"threads_started":1}
I20260620 02:33:25.293236 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.277389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:42960)
0620 02:33:25.280919 (+ 3530us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.280979 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:25.281300 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:25.282334 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:25.282342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:25.282379 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:25.283431 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.283439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.285363 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.285368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.288106 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.288127 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.290525 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.290531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.290747 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.290779 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:33:25.292690 (+ 1911us) client_negotiation.cc:240] Negotiation successful
0620 02:33:25.292847 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3226,"mutex_wait_us":69,"spinlock_wait_cycles":2816,"thread_start_us":4289,"threads_started":1}
I20260620 02:33:25.295219 18516 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0d73c1ed0c43069f306c0d4e888a1d (DEFAULT_TABLE table=test-table [id=452682d84aab49b99ffca3a8bf4d8b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:25.295480 18386 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0d73c1ed0c43069f306c0d4e888a1d (DEFAULT_TABLE table=test-table [id=452682d84aab49b99ffca3a8bf4d8b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:25.296885 18516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0d73c1ed0c43069f306c0d4e888a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:25.296911 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0d73c1ed0c43069f306c0d4e888a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:25.299280 18255 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0d73c1ed0c43069f306c0d4e888a1d (DEFAULT_TABLE table=test-table [id=452682d84aab49b99ffca3a8bf4d8b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:25.300865 18255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0d73c1ed0c43069f306c0d4e888a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:25.308954 18600 tablet_bootstrap.cc:492] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Bootstrap starting.
I20260620 02:33:25.312923 18600 tablet_bootstrap.cc:654] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:25.313949 18601 tablet_bootstrap.cc:492] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Bootstrap starting.
I20260620 02:33:25.314389 18600 log.cc:824] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:25.317811 18601 tablet_bootstrap.cc:654] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:25.318037 18600 tablet_bootstrap.cc:492] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: No bootstrap required, opened a new log
I20260620 02:33:25.318344 18600 ts_tablet_manager.cc:1397] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260620 02:33:25.319039 18603 tablet_bootstrap.cc:492] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Bootstrap starting.
I20260620 02:33:25.319176 18601 log.cc:824] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:25.322875 18603 tablet_bootstrap.cc:654] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:25.323920 18601 tablet_bootstrap.cc:492] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: No bootstrap required, opened a new log
I20260620 02:33:25.323953 18603 log.cc:824] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:25.324246 18601 ts_tablet_manager.cc:1397] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 02:33:25.326936 18603 tablet_bootstrap.cc:492] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: No bootstrap required, opened a new log
I20260620 02:33:25.327222 18603 ts_tablet_manager.cc:1397] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 02:33:25.329069 18600 raft_consensus.cc:348] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.329563 18600 raft_consensus.cc:374] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:25.329672 18600 raft_consensus.cc:729] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701230de4dba46dfa39a691e1314b5aa, State: Initialized, Role: FOLLOWER
I20260620 02:33:25.330355 18600 consensus_queue.cc:260] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.332698 18581 heartbeater.cc:503] Master 127.0.97.62:37689 was elected leader, sending a full tablet report...
I20260620 02:33:25.333575 18600 ts_tablet_manager.cc:1428] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
I20260620 02:33:25.335421 18601 raft_consensus.cc:348] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.335908 18601 raft_consensus.cc:374] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:25.336030 18601 raft_consensus.cc:729] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822525958d842be9b74babd12539d44, State: Initialized, Role: FOLLOWER
I20260620 02:33:25.336781 18601 consensus_queue.cc:260] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.337145 18603 raft_consensus.cc:348] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.337751 18603 raft_consensus.cc:374] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:25.337940 18603 raft_consensus.cc:729] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61fc9723955d47079fb1e90d1658f4cd, State: Initialized, Role: FOLLOWER
I20260620 02:33:25.338060 18451 heartbeater.cc:503] Master 127.0.97.62:37689 was elected leader, sending a full tablet report...
I20260620 02:33:25.338899 18601 ts_tablet_manager.cc:1428] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 02:33:25.338742 18603 consensus_queue.cc:260] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.341074 18603 ts_tablet_manager.cc:1428] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260620 02:33:25.422331 18452 tablet.cc:2369] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:25.442010 18606 raft_consensus.cc:482] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:25.442414 18606 raft_consensus.cc:504] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.444306 18606 leader_election.cc:288] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e822525958d842be9b74babd12539d44 (127.0.97.2:43793), 61fc9723955d47079fb1e90d1658f4cd (127.0.97.1:40593)
W20260620 02:33:25.444635 18321 tablet.cc:2369] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:25.450474 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.444787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54599 (local address 127.0.97.1:40593)
0620 02:33:25.445029 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.445036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.445534 (+ 498us) server_negotiation.cc:407] Connection header received
0620 02:33:25.445627 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.445631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.445691 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.445800 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:25.446263 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.446983 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.448671 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.449525 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.449664 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.449767 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.450073 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.450216 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:33:25.450525 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.444612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.97.3:54599)
0620 02:33:25.445199 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.445233 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:25.445391 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:25.445840 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:25.445845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:25.445926 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:25.446088 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.446096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.447121 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.447128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.448497 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.448509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.449690 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.449696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.449895 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.449920 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:33:25.450044 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:33:25.450188 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":2,"spinlock_wait_cycles":21760}
W20260620 02:33:25.450554 18582 tablet.cc:2369] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:25.451526 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.444612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43793 (local address 127.0.97.3:57259)
0620 02:33:25.445704 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.445718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:25.445885 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:25.446319 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:25.446323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:25.446345 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:25.446670 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.446678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.447866 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.447878 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.449306 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:25.449319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.450676 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.450683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:25.450879 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.450920 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:33:25.451025 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 02:33:25.451151 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":209,"thread_start_us":551,"threads_started":1}
I20260620 02:33:25.451581 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.444899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57259 (local address 127.0.97.2:43793)
0620 02:33:25.445155 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.445161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.445816 (+ 655us) server_negotiation.cc:407] Connection header received
0620 02:33:25.446071 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.446094 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.446197 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.446302 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:25.446824 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.447712 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.449494 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.450325 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.450455 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.450556 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.451160 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.451312 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:33:25.451911 18275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0d73c1ed0c43069f306c0d4e888a1d" candidate_uuid: "701230de4dba46dfa39a691e1314b5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61fc9723955d47079fb1e90d1658f4cd" is_pre_election: true
I20260620 02:33:25.452503 18275 raft_consensus.cc:2454] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 701230de4dba46dfa39a691e1314b5aa in term 0.
I20260620 02:33:25.452425 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0d73c1ed0c43069f306c0d4e888a1d" candidate_uuid: "701230de4dba46dfa39a691e1314b5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e822525958d842be9b74babd12539d44" is_pre_election: true
I20260620 02:33:25.452944 18406 raft_consensus.cc:2454] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 701230de4dba46dfa39a691e1314b5aa in term 0.
I20260620 02:33:25.453296 18472 leader_election.cc:302] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61fc9723955d47079fb1e90d1658f4cd, 701230de4dba46dfa39a691e1314b5aa; no voters:
I20260620 02:33:25.453768 18606 raft_consensus.cc:2781] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:33:25.453919 18606 raft_consensus.cc:482] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:33:25.454031 18606 raft_consensus.cc:3037] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:25.456233 18606 raft_consensus.cc:504] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.456831 18606 leader_election.cc:288] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [CANDIDATE]: Term 1 election: Requested vote from peers e822525958d842be9b74babd12539d44 (127.0.97.2:43793), 61fc9723955d47079fb1e90d1658f4cd (127.0.97.1:40593)
I20260620 02:33:25.457389 18275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0d73c1ed0c43069f306c0d4e888a1d" candidate_uuid: "701230de4dba46dfa39a691e1314b5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61fc9723955d47079fb1e90d1658f4cd"
I20260620 02:33:25.457381 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0d73c1ed0c43069f306c0d4e888a1d" candidate_uuid: "701230de4dba46dfa39a691e1314b5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e822525958d842be9b74babd12539d44"
I20260620 02:33:25.457607 18275 raft_consensus.cc:3037] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:25.457633 18406 raft_consensus.cc:3037] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:25.459849 18406 raft_consensus.cc:2454] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 701230de4dba46dfa39a691e1314b5aa in term 1.
I20260620 02:33:25.459851 18275 raft_consensus.cc:2454] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 701230de4dba46dfa39a691e1314b5aa in term 1.
I20260620 02:33:25.460330 18469 leader_election.cc:302] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701230de4dba46dfa39a691e1314b5aa, e822525958d842be9b74babd12539d44; no voters:
I20260620 02:33:25.460654 18606 raft_consensus.cc:2781] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:25.461189 18606 raft_consensus.cc:686] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 1 LEADER]: Becoming Leader. State: Replica: 701230de4dba46dfa39a691e1314b5aa, State: Running, Role: LEADER
I20260620 02:33:25.461670 18606 consensus_queue.cc:237] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } }
I20260620 02:33:25.467963 18133 catalog_manager.cc:5495] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 701230de4dba46dfa39a691e1314b5aa (127.0.97.3). New cstate: current_term: 1 leader_uuid: "701230de4dba46dfa39a691e1314b5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "701230de4dba46dfa39a691e1314b5aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37661 } health_report { overall_health: HEALTHY } } }
I20260620 02:33:25.486248 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.479992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.0.97.3:37661)
0620 02:33:25.480184 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.480190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.480334 (+ 144us) server_negotiation.cc:407] Connection header received
0620 02:33:25.480702 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.480707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.480798 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.480945 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:33:25.481451 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.482201 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.483851 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.484277 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.484385 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.484896 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:33:25.485527 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:33:25.485671 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.485829 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.485936 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:33:25.494788 18275 raft_consensus.cc:1261] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 1 FOLLOWER]: Refusing update from remote peer 701230de4dba46dfa39a691e1314b5aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:25.494825 18406 raft_consensus.cc:1261] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 1 FOLLOWER]: Refusing update from remote peer 701230de4dba46dfa39a691e1314b5aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:25.495810 18606 consensus_queue.cc:1035] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e822525958d842be9b74babd12539d44" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:25.496114 18610 consensus_queue.cc:1035] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "61fc9723955d47079fb1e90d1658f4cd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:25.509927 18615 mvcc.cc:205] Tried to move back new op lower bound from 7298755811296137216 to 7298755811176194048. Current Snapshot: MvccSnapshot[applied={T|T < 7298755811296137216}]
I20260620 02:33:25.509905 18614 mvcc.cc:205] Tried to move back new op lower bound from 7298755811296137216 to 7298755811176194048. Current Snapshot: MvccSnapshot[applied={T|T < 7298755811296137216}]
I20260620 02:33:25.533797 18133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:25.534116 18133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:25.537858 18133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f18780ca6c3452194591885183ea102: Sending DeleteTablet for 3 replicas of tablet 7b0d73c1ed0c43069f306c0d4e888a1d
I20260620 02:33:25.539151 18386 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0d73c1ed0c43069f306c0d4e888a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:25 UTC) from {username='slave'} at 127.0.0.1:56978
I20260620 02:33:25.539216 18255 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0d73c1ed0c43069f306c0d4e888a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:25 UTC) from {username='slave'} at 127.0.0.1:42960
I20260620 02:33:25.539665 18516 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0d73c1ed0c43069f306c0d4e888a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:25 UTC) from {username='slave'} at 127.0.0.1:48104
I20260620 02:33:25.542198 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18193
I20260620 02:33:25.548574 18626 tablet_replica.cc:330] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: stopping tablet replica
I20260620 02:33:25.548671 18624 tablet_replica.cc:330] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: stopping tablet replica
I20260620 02:33:25.549199 18624 raft_consensus.cc:2229] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:25.549223 18626 raft_consensus.cc:2229] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:33:25.549746 18624 raft_consensus.cc:2258] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:25.550556 18625 tablet_replica.cc:330] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: stopping tablet replica
I20260620 02:33:25.551051 18625 raft_consensus.cc:2229] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:25.551613 18625 raft_consensus.cc:2258] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:25.552151 18624 ts_tablet_manager.cc:1905] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:25.550045 18626 raft_consensus.cc:2258] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:25.553648 18625 ts_tablet_manager.cc:1905] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:25.559999 18624 ts_tablet_manager.cc:1918] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:25.560269 18624 log.cc:1196] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/7b0d73c1ed0c43069f306c0d4e888a1d
I20260620 02:33:25.560900 18624 ts_tablet_manager.cc:1939] T 7b0d73c1ed0c43069f306c0d4e888a1d P e822525958d842be9b74babd12539d44: Deleting consensus metadata
I20260620 02:33:25.561355 18625 ts_tablet_manager.cc:1918] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:25.562973 18626 ts_tablet_manager.cc:1905] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:25.563004 18118 catalog_manager.cc:4841] TS e822525958d842be9b74babd12539d44 (127.0.97.2:43793): tablet 7b0d73c1ed0c43069f306c0d4e888a1d (table test-table [id=452682d84aab49b99ffca3a8bf4d8b46]) successfully deleted
I20260620 02:33:25.566825 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.543363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.0.97.1:40593)
0620 02:33:25.543659 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.543669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.557914 (+ 14245us) server_negotiation.cc:407] Connection header received
0620 02:33:25.558429 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.558436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.558577 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.558666 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:25.559350 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.560532 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.562642 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.563292 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.563396 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.564988 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:25.565063 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:25.565096 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:25.565887 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:25.565949 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:25.565983 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:25.566099 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.566281 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.566400 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 02:33:25.569653 18626 ts_tablet_manager.cc:1918] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:25.569854 18626 log.cc:1196] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/7b0d73c1ed0c43069f306c0d4e888a1d
I20260620 02:33:25.570070 18315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:25.570118 18625 log.cc:1196] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/7b0d73c1ed0c43069f306c0d4e888a1d
I20260620 02:33:25.570395 18626 ts_tablet_manager.cc:1939] T 7b0d73c1ed0c43069f306c0d4e888a1d P 701230de4dba46dfa39a691e1314b5aa: Deleting consensus metadata
I20260620 02:33:25.572374 18119 catalog_manager.cc:4841] TS 701230de4dba46dfa39a691e1314b5aa (127.0.97.3:37661): tablet 7b0d73c1ed0c43069f306c0d4e888a1d (table test-table [id=452682d84aab49b99ffca3a8bf4d8b46]) successfully deleted
I20260620 02:33:25.678272 18625 ts_tablet_manager.cc:1939] T 7b0d73c1ed0c43069f306c0d4e888a1d P 61fc9723955d47079fb1e90d1658f4cd: Deleting consensus metadata
I20260620 02:33:25.678370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18193
I20260620 02:33:25.706305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18324
W20260620 02:33:25.706349 18121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40593 (error 108)
I20260620 02:33:25.709635 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.709092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:42986)
0620 02:33:25.709288 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.709426 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 02:33:25.710004 18121 catalog_manager.cc:4584] TS 61fc9723955d47079fb1e90d1658f4cd (127.0.97.1:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b0d73c1ed0c43069f306c0d4e888a1d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
I20260620 02:33:25.715219 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.707696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.0.97.2:43793)
0620 02:33:25.707885 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.707891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.707915 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:33:25.707988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.707994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.708139 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.708254 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:25.708978 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.710087 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.712009 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.712439 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.712539 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.713832 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:25.713840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:25.713854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:25.714450 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:25.714496 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:25.714515 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:25.714612 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.714766 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.714857 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:33:25.715960 18446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:25.763969 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.763467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43002)
0620 02:33:25.763716 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.763802 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 02:33:25.765115 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.764704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43008)
0620 02:33:25.764862 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.764943 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 02:33:25.825659 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.825168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43012)
0620 02:33:25.825382 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.825469 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 02:33:25.827356 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.826972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43014)
0620 02:33:25.827130 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.827237 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 02:33:25.865145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18324
I20260620 02:33:25.889540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18454
I20260620 02:33:25.897921 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.890413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.0.97.3:37661)
0620 02:33:25.890601 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:25.890606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:25.890636 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:33:25.890893 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:25.890897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:25.890983 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:25.891100 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:25.891584 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.892377 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.894480 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:25.894915 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:25.895067 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:25.896275 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:25.896287 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:25.896301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:25.896914 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:25.896967 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:25.896986 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:25.897154 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:25.897400 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 02:33:25.897511 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:33:25.898732 18576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:25.907963 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.907548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43030)
0620 02:33:25.907711 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.907798 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:33:25.909197 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:25.908708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43044)
0620 02:33:25.908908 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:25.908986 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 02:33:26.023964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18454
I20260620 02:33:26.045046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18103
I20260620 02:33:26.046272 18164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:26.158164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18103
I20260620 02:33:26.158284 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:26.157714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40593 (local address 127.0.0.1:43050)
0620 02:33:26.157963 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:26.158081 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
2026-06-20T02:33:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3113 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:26 dist-test-slave-r706 krb5kdc[18635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:26 dist-test-slave-r706 krb5kdc[18635](info): set up 2 sockets
Jun 20 02:33:26 dist-test-slave-r706 krb5kdc[18635](info): commencing operation
krb5kdc: starting...
W20260620 02:33:27.309368 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:27 dist-test-slave-r706 krb5kdc[18635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:33:27.613330 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46403
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:27.965958 18648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:27.966288 18648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:27.966390 18648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:27.973870 18648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:27.973999 18648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:27.974093 18648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:27.974205 18648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:27.974273 18648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:27.984705 18648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:27.986187 18648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:27.987588 18648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:28.000905 18656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:28.000948 18654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:28.000905 18653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:28.002272 18648 server_base.cc:1034] running on GCE node
I20260620 02:33:28.003222 18648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:28.004806 18648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:28.006065 18648 hybrid_clock.cc:648] HybridClock initialized: now 1781922808006025 us; error 67 us; skew 500 ppm
I20260620 02:33:28.006448 18648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:28 dist-test-slave-r706 krb5kdc[18635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:28.009799 18648 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:33:28.011528 18648 webserver.cc:460] Webserver started at http://127.0.97.62:38835/ using document root <none> and password file <none>
I20260620 02:33:28.012259 18648 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:28.012421 18648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:28.012761 18648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:28.015918 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "7fcbc624f31e42bc8243c55a8201fcb4"
format_stamp: "Formatted at 2026-06-20 02:33:28 on dist-test-slave-r706"
server_key: "f836054908441b2e0986ac058f03e0e2"
server_key_iv: "aa799df4a17210fe91fcaafb06ee7ad3"
server_key_version: "encryptionkey@0"
I20260620 02:33:28.016710 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "7fcbc624f31e42bc8243c55a8201fcb4"
format_stamp: "Formatted at 2026-06-20 02:33:28 on dist-test-slave-r706"
server_key: "f836054908441b2e0986ac058f03e0e2"
server_key_iv: "aa799df4a17210fe91fcaafb06ee7ad3"
server_key_version: "encryptionkey@0"
I20260620 02:33:28.022392 18648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:33:28.026283 18663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:28.027693 18648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:33:28.027901 18648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "7fcbc624f31e42bc8243c55a8201fcb4"
format_stamp: "Formatted at 2026-06-20 02:33:28 on dist-test-slave-r706"
server_key: "f836054908441b2e0986ac058f03e0e2"
server_key_iv: "aa799df4a17210fe91fcaafb06ee7ad3"
server_key_version: "encryptionkey@0"
I20260620 02:33:28.028126 18648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:28.048834 18648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:28.053722 18648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:28.054073 18648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:28.074100 18648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46403
I20260620 02:33:28.074142 18714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46403 every 8 connection(s)
I20260620 02:33:28.075529 18648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:28.079367 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:28.083567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18648
I20260620 02:33:28.083776 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:28.084319 388 external_mini_cluster.cc:1428] Setting key d21c2f63226e310423ac862fa529cac8
I20260620 02:33:28.093667 18715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4: Bootstrap starting.
I20260620 02:33:28.097759 18715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:28.099187 18715 log.cc:824] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:33:28 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:28.102798 18715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4: No bootstrap required, opened a new log
I20260620 02:33:28.111120 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:28.086421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.0.97.62:46403)
0620 02:33:28.087276 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:28.087287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:28.087331 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:33:28.089185 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:28.089210 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:28.089666 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:28.090032 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:28.091989 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:28.093441 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:28.095323 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:28.096137 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:28.096294 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:28.101886 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:28.101909 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:28.101927 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:28.101975 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:28.106077 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:28.106899 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:28.106910 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:28.106920 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:28.107167 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:28.107629 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:28.107634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:28.107638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:28.108409 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:28.108742 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:28.109169 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 02:33:28.109656 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":417,"threads_started":1}
I20260620 02:33:28.113808 18715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46403 } }
I20260620 02:33:28.114256 18715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:28.114370 18715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fcbc624f31e42bc8243c55a8201fcb4, State: Initialized, Role: FOLLOWER
I20260620 02:33:28.114924 18715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46403 } }
I20260620 02:33:28.115159 18715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:28.115316 18715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:28.115483 18715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:28.117995 18715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46403 } }
I20260620 02:33:28.118530 18715 leader_election.cc:302] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fcbc624f31e42bc8243c55a8201fcb4; no voters:
I20260620 02:33:28.119131 18715 leader_election.cc:288] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:28.119315 18720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:28.120186 18720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [term 1 LEADER]: Becoming Leader. State: Replica: 7fcbc624f31e42bc8243c55a8201fcb4, State: Running, Role: LEADER
I20260620 02:33:28.120837 18720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46403 } }
I20260620 02:33:28.121326 18715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:28.124727 18721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46403 } } }
I20260620 02:33:28.125281 18722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fcbc624f31e42bc8243c55a8201fcb4. Latest consensus state: current_term: 1 leader_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcbc624f31e42bc8243c55a8201fcb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46403 } } }
I20260620 02:33:28.125694 18721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:28.125694 18722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:28.129964 18730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:28.137916 18730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:28.148130 18730 catalog_manager.cc:1347] Generated new cluster ID: 0e2262ff1bbd4070a3f304f61849c4b1
I20260620 02:33:28.148273 18730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:28.162674 18730 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:28.164112 18730 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:28.172158 18730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4: Generated new TSK 0
I20260620 02:33:28.173224 18730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:33:28.253782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46403
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:28.605997 18741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:28.606346 18741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:28.606518 18741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:28.614104 18741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:28.614259 18741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:28.614434 18741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:28.624830 18741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:28.626166 18741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:28.627460 18741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:28.641680 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:28.641656 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:28.641933 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:28.643415 18741 server_base.cc:1034] running on GCE node
I20260620 02:33:28.644217 18741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:28.645364 18741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:28.646622 18741 hybrid_clock.cc:648] HybridClock initialized: now 1781922808646566 us; error 34 us; skew 500 ppm
I20260620 02:33:28.647023 18741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:28 dist-test-slave-r706 krb5kdc[18635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:28.655748 18741 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:33:28.657541 18741 webserver.cc:460] Webserver started at http://127.0.97.1:45727/ using document root <none> and password file <none>
I20260620 02:33:28.658324 18741 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:28.658476 18741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:28.658802 18741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:28.661983 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "166ce3994c93477790f50e73c4849c74"
format_stamp: "Formatted at 2026-06-20 02:33:28 on dist-test-slave-r706"
server_key: "9b6cc9c0791d3dd7119cc3b5e8e5e019"
server_key_iv: "e1474bc3dee489784a5e76e2e387ab99"
server_key_version: "encryptionkey@0"
I20260620 02:33:28.662855 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "166ce3994c93477790f50e73c4849c74"
format_stamp: "Formatted at 2026-06-20 02:33:28 on dist-test-slave-r706"
server_key: "9b6cc9c0791d3dd7119cc3b5e8e5e019"
server_key_iv: "e1474bc3dee489784a5e76e2e387ab99"
server_key_version: "encryptionkey@0"
I20260620 02:33:28.668497 18741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:33:28.672324 18756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:28.673851 18741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:33:28.674054 18741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "166ce3994c93477790f50e73c4849c74"
format_stamp: "Formatted at 2026-06-20 02:33:28 on dist-test-slave-r706"
server_key: "9b6cc9c0791d3dd7119cc3b5e8e5e019"
server_key_iv: "e1474bc3dee489784a5e76e2e387ab99"
server_key_version: "encryptionkey@0"
I20260620 02:33:28.674278 18741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:28.698001 18741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:28.702767 18741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:28.703032 18741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:28.704074 18741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:28.705565 18741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:28.705648 18741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:28.705735 18741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:28.705780 18741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:28.741873 18741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41157
I20260620 02:33:28.741940 18868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41157 every 8 connection(s)
I20260620 02:33:28.743250 18741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:28.745814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18741
I20260620 02:33:28.746013 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:33:28.746547 388 external_mini_cluster.cc:1428] Setting key b146e3ea533717fd3bb6e99fc2cfca33
Jun 20 02:33:28 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:28.766057 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:28.746108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44619 (local address 127.0.97.62:46403)
0620 02:33:28.746762 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:28.746770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:28.747761 (+ 991us) server_negotiation.cc:407] Connection header received
0620 02:33:28.749859 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:28.749864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:28.749955 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:28.750088 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:28.751690 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:28.752446 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:28.754153 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:28.754605 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:28.754758 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:28.759191 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:28.759197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:28.759200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:28.759239 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:28.762160 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:28.763335 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:28.763339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:28.763342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:28.763472 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:28.764019 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:28.764022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:28.764024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:28.764309 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:28.764426 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:28.765557 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0620 02:33:28.765759 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":256,"threads_started":1}
I20260620 02:33:28.767665 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:28.746681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46403 (local address 127.0.97.1:44619)
0620 02:33:28.747553 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:28.747616 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:28.749572 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:28.750353 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:28.750362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:28.750920 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:28.751500 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:28.751522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:28.752643 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:28.752647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:28.753994 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:28.754007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:28.754862 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:28.754872 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:28.755132 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:28.755892 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:28.755917 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:28.758870 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:28.762361 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:28.762375 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:28.762410 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:28.763188 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:28.763589 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:28.763597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:28.763603 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:28.763886 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:28.764460 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:28.764472 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:28.765037 (+ 565us) client_negotiation.cc:769] Sending connection context
0620 02:33:28.765395 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 02:33:28.766049 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":382,"threads_started":1}
I20260620 02:33:28.769452 18869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46403
I20260620 02:33:28.769811 18869 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:28.770591 18869 heartbeater.cc:511] Master 127.0.97.62:46403 requested a full tablet report, sending...
I20260620 02:33:28.772859 18680 ts_manager.cc:194] Registered new tserver with Master: 166ce3994c93477790f50e73c4849c74 (127.0.97.1:41157)
I20260620 02:33:28.774356 18680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44619
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:33:28.812081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46403
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:29.153136 18875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:29.153483 18875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:29.153653 18875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:29.161195 18875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:29.161351 18875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:29.161538 18875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:29.171660 18875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:29.173249 18875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:29.174556 18875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:29.187620 18880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:29.187668 18883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:29.187745 18881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:29.188949 18875 server_base.cc:1034] running on GCE node
I20260620 02:33:29.189620 18875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:29.190584 18875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:29.191833 18875 hybrid_clock.cc:648] HybridClock initialized: now 1781922809191795 us; error 51 us; skew 500 ppm
I20260620 02:33:29.192188 18875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:29 dist-test-slave-r706 krb5kdc[18635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:29.195675 18875 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:33:29.197288 18875 webserver.cc:460] Webserver started at http://127.0.97.2:34601/ using document root <none> and password file <none>
I20260620 02:33:29.198009 18875 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:29.198159 18875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:29.198487 18875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:29.201447 18875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "233b9f79fb41468485956acb2401cd71"
format_stamp: "Formatted at 2026-06-20 02:33:29 on dist-test-slave-r706"
server_key: "f4d3e0979ee4a09f12b2211867df42d4"
server_key_iv: "0bedbfde4ea0047a6df39b455c7e8577"
server_key_version: "encryptionkey@0"
I20260620 02:33:29.202255 18875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "233b9f79fb41468485956acb2401cd71"
format_stamp: "Formatted at 2026-06-20 02:33:29 on dist-test-slave-r706"
server_key: "f4d3e0979ee4a09f12b2211867df42d4"
server_key_iv: "0bedbfde4ea0047a6df39b455c7e8577"
server_key_version: "encryptionkey@0"
I20260620 02:33:29.207293 18875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:33:29.210851 18890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:29.212136 18875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:33:29.212311 18875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "233b9f79fb41468485956acb2401cd71"
format_stamp: "Formatted at 2026-06-20 02:33:29 on dist-test-slave-r706"
server_key: "f4d3e0979ee4a09f12b2211867df42d4"
server_key_iv: "0bedbfde4ea0047a6df39b455c7e8577"
server_key_version: "encryptionkey@0"
I20260620 02:33:29.212525 18875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:29.231909 18875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:29.236533 18875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:29.236847 18875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:29.237917 18875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:29.239387 18875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:29.239509 18875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:29.239660 18875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:29.239738 18875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:29.274695 18875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40329
I20260620 02:33:29.274746 19002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40329 every 8 connection(s)
I20260620 02:33:29.276077 18875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:29.281237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 18875
I20260620 02:33:29.281642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:33:29.282423 388 external_mini_cluster.cc:1428] Setting key def9cabdb4ce8ab538980b324df568fe
Jun 20 02:33:29 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:29.300884 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.278969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39887 (local address 127.0.97.62:46403)
0620 02:33:29.279562 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:29.279572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:29.282256 (+ 2684us) server_negotiation.cc:407] Connection header received
0620 02:33:29.283688 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:29.283693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:29.283764 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:29.283882 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:29.285875 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.287155 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.288778 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.289560 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.289714 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.293841 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:29.293851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:29.293859 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:29.293911 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:29.297080 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:29.298150 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:29.298154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:29.298156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:29.298294 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:29.298907 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:29.298911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:29.298913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:29.299220 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:29.299347 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:29.300380 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0620 02:33:29.300595 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":281,"threads_started":1}
I20260620 02:33:29.302457 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.279643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46403 (local address 127.0.97.2:39887)
0620 02:33:29.280718 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:29.280776 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:29.283333 (+ 2557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:29.284202 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:29.284215 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:29.285048 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:29.285674 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.285696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.287324 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.287328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.288620 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.288632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.289791 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.289796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.290021 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.290779 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:29.290805 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:29.293567 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:29.297243 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:29.297257 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:29.297280 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:29.298014 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:29.298410 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:29.298417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:29.298423 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:29.298734 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:29.299366 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:29.299377 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:29.299922 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 02:33:29.300255 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 02:33:29.300883 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":434,"threads_started":1}
I20260620 02:33:29.304104 19003 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46403
I20260620 02:33:29.304438 19003 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:29.305267 19003 heartbeater.cc:511] Master 127.0.97.62:46403 requested a full tablet report, sending...
I20260620 02:33:29.306828 18680 ts_manager.cc:194] Registered new tserver with Master: 233b9f79fb41468485956acb2401cd71 (127.0.97.2:40329)
I20260620 02:33:29.307701 18680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39887
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:33:29.349668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46403
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:29.682525 19009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:29.682842 19009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:29.682940 19009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:29.690376 19009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:29.690503 19009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:29.690623 19009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:29.700834 19009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:29.702154 19009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:29.703465 19009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:29.717772 19014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:29.717751 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:29.717722 19015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:29.718798 19009 server_base.cc:1034] running on GCE node
I20260620 02:33:29.719532 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:29.720566 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:29.721804 19009 hybrid_clock.cc:648] HybridClock initialized: now 1781922809721758 us; error 74 us; skew 500 ppm
I20260620 02:33:29.722184 19009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:29 dist-test-slave-r706 krb5kdc[18635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:29.725734 19009 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:33:29.727290 19009 webserver.cc:460] Webserver started at http://127.0.97.3:40593/ using document root <none> and password file <none>
I20260620 02:33:29.728014 19009 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:29.728142 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:29.728468 19009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:29.731503 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "99b08e3d28394638b2a679c07da26eaa"
format_stamp: "Formatted at 2026-06-20 02:33:29 on dist-test-slave-r706"
server_key: "e17b1370565ea2e0384e62c091d8069a"
server_key_iv: "eb13667c04b31863eed689f9b9e2b1d0"
server_key_version: "encryptionkey@0"
I20260620 02:33:29.732316 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "99b08e3d28394638b2a679c07da26eaa"
format_stamp: "Formatted at 2026-06-20 02:33:29 on dist-test-slave-r706"
server_key: "e17b1370565ea2e0384e62c091d8069a"
server_key_iv: "eb13667c04b31863eed689f9b9e2b1d0"
server_key_version: "encryptionkey@0"
I20260620 02:33:29.737630 19009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 02:33:29.741264 19024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:29.742985 19009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:29.743163 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "99b08e3d28394638b2a679c07da26eaa"
format_stamp: "Formatted at 2026-06-20 02:33:29 on dist-test-slave-r706"
server_key: "e17b1370565ea2e0384e62c091d8069a"
server_key_iv: "eb13667c04b31863eed689f9b9e2b1d0"
server_key_version: "encryptionkey@0"
I20260620 02:33:29.743402 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:29.757664 19009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:29.762179 19009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:29.762458 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:29.763437 19009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:29.764807 19009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:29.764887 19009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:29.764976 19009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:29.765097 19009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 02:33:29.778007 18869 heartbeater.cc:503] Master 127.0.97.62:46403 was elected leader, sending a full tablet report...
I20260620 02:33:29.802403 19009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44407
I20260620 02:33:29.802503 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44407 every 8 connection(s)
I20260620 02:33:29.803781 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:29.808275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19009
I20260620 02:33:29.808449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:33:29.808903 388 external_mini_cluster.cc:1428] Setting key cb51395a7c7488ca126448eabbf22cb0
Jun 20 02:33:29 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:29.825749 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.806419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44867 (local address 127.0.97.62:46403)
0620 02:33:29.806842 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:29.806847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:29.807799 (+ 952us) server_negotiation.cc:407] Connection header received
0620 02:33:29.810036 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:29.810040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:29.810140 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:29.810256 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:29.812249 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.813093 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.814588 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.814986 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.815083 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.819119 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:29.819125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:29.819129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:29.819168 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:29.822145 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:29.823161 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:29.823166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:29.823195 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:29.823390 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:29.823913 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:29.823918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:29.823922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:29.824270 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:29.824451 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:29.825259 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 02:33:29.825460 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":197,"threads_started":1}
I20260620 02:33:29.826725 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.806946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46403 (local address 127.0.97.3:44867)
0620 02:33:29.807638 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:29.807674 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:29.809709 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:29.810508 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:29.810517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:29.811280 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:29.812091 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.812112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.813226 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.813229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.814460 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.814472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.815249 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.815255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.815511 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.816327 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:29.816385 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:29.818843 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:29.822374 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:29.822382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:29.822397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:29.823025 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:29.823511 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:29.823514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:29.823516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:29.823797 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:29.824438 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:29.824451 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:29.824824 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 02:33:29.825149 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 02:33:29.825596 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":276,"threads_started":1}
I20260620 02:33:29.828212 19137 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46403
I20260620 02:33:29.828588 19137 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:29.829392 19137 heartbeater.cc:511] Master 127.0.97.62:46403 requested a full tablet report, sending...
I20260620 02:33:29.830922 18680 ts_manager.cc:194] Registered new tserver with Master: 99b08e3d28394638b2a679c07da26eaa (127.0.97.3:44407)
I20260620 02:33:29.831763 18680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44867
I20260620 02:33:29.837136 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:33:29.855736 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.841275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.0.97.62:46403)
0620 02:33:29.841487 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:29.841495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:29.841925 (+ 430us) server_negotiation.cc:407] Connection header received
0620 02:33:29.842130 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:29.842135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:29.842214 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:29.842324 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:29.843834 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.844892 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.846353 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.846765 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.846912 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.850043 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:29.850048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:29.850051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:29.850088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:29.852626 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:29.853571 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:29.853577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:29.853581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:29.853716 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:29.854203 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:29.854207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:29.854210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:29.854536 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:29.854825 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:29.855219 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 02:33:29.855428 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:33:29.860018 18680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:33:29.866972 18680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:33:29.892922 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.880111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.97.1:41157)
0620 02:33:29.883993 (+ 3882us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:29.883999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:29.884024 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:33:29.884243 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:29.884262 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:29.885087 (+ 825us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:29.885346 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:29.886259 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.887939 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.890394 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.891591 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.891758 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.891994 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:29.892383 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 02:33:29.892587 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3718,"thread_start_us":3579,"threads_started":1}
I20260620 02:33:29.892993 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.879798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41157 (local address 127.0.0.1:36770)
0620 02:33:29.881710 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:29.881749 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:29.881939 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:29.885368 (+ 3429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:29.885385 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:29.885427 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:29.886033 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.886048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.888109 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.888116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.890230 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.890242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.891846 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.891852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.892096 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.892158 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 02:33:29.892301 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:33:29.892510 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"mutex_wait_us":191,"thread_start_us":1538,"threads_started":1}
I20260620 02:33:29.894129 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.880752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.0.97.2:40329)
0620 02:33:29.882954 (+ 2202us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:29.882964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:29.884588 (+ 1624us) server_negotiation.cc:407] Connection header received
0620 02:33:29.884856 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:29.884863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:29.885075 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:29.885331 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:29.886470 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.888025 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.891112 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.892447 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.892584 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.892753 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:29.893685 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 02:33:29.893846 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2017,"thread_start_us":1921,"threads_started":1}
I20260620 02:33:29.894160 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.880134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40329 (local address 127.0.0.1:34742)
0620 02:33:29.884449 (+ 4315us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:29.884478 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:29.884659 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:29.885594 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:29.885604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:29.885623 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:29.886246 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.886259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.888166 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.888170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.890940 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.890954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.893315 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.893321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.893492 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.893515 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:33:29.893623 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:33:29.893754 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4152,"mutex_wait_us":88,"spinlock_wait_cycles":9856,"thread_start_us":2998,"threads_started":1}
I20260620 02:33:29.895306 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.882735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44407 (local address 127.0.0.1:47562)
0620 02:33:29.884492 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:29.884513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:29.884666 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:29.885898 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:29.885904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:29.885934 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:29.886477 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.886494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.888303 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.888309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.890940 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:29.890954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.894524 (+ 3570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.894532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:29.894707 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.894738 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:33:29.894804 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 02:33:29.894942 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1627,"mutex_wait_us":182,"spinlock_wait_cycles":27904,"thread_start_us":888,"threads_started":1}
I20260620 02:33:29.896342 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:29.883331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.0.97.3:44407)
0620 02:33:29.885343 (+ 2012us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:29.885351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:29.885383 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:33:29.885480 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:29.885487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:29.885691 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:29.885910 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:29.886647 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.888140 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.892849 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:29.894298 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:29.895544 (+ 1246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:29.895752 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:29.895825 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 02:33:29.895999 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":1655,"threads_started":1}
I20260620 02:33:29.897269 18804 tablet_service.cc:1467] Processing CreateTablet for tablet 4a858b4efa884533966b41b406ef857a (DEFAULT_TABLE table=test-table [id=58eb28099023462b9d11a132daad8a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:29.897527 18938 tablet_service.cc:1467] Processing CreateTablet for tablet 4a858b4efa884533966b41b406ef857a (DEFAULT_TABLE table=test-table [id=58eb28099023462b9d11a132daad8a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:29.898885 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a858b4efa884533966b41b406ef857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:29.898960 18938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a858b4efa884533966b41b406ef857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:29.901190 19072 tablet_service.cc:1467] Processing CreateTablet for tablet 4a858b4efa884533966b41b406ef857a (DEFAULT_TABLE table=test-table [id=58eb28099023462b9d11a132daad8a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:29.902706 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a858b4efa884533966b41b406ef857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:29.912763 19156 tablet_bootstrap.cc:492] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Bootstrap starting.
I20260620 02:33:29.913678 19157 tablet_bootstrap.cc:492] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Bootstrap starting.
I20260620 02:33:29.916646 19156 tablet_bootstrap.cc:654] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:29.917094 19158 tablet_bootstrap.cc:492] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Bootstrap starting.
I20260620 02:33:29.917194 19157 tablet_bootstrap.cc:654] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:29.918182 19156 log.cc:824] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:29.918433 19157 log.cc:824] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:29.920583 19158 tablet_bootstrap.cc:654] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:29.921397 19156 tablet_bootstrap.cc:492] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: No bootstrap required, opened a new log
I20260620 02:33:29.921553 19157 tablet_bootstrap.cc:492] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: No bootstrap required, opened a new log
I20260620 02:33:29.921684 19156 ts_tablet_manager.cc:1397] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:33:29.921814 19157 ts_tablet_manager.cc:1397] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 02:33:29.922034 19158 log.cc:824] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:29.925403 19158 tablet_bootstrap.cc:492] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: No bootstrap required, opened a new log
I20260620 02:33:29.925709 19158 ts_tablet_manager.cc:1397] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:33:29.929903 19156 raft_consensus.cc:348] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:29.930310 19156 raft_consensus.cc:374] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:29.930428 19156 raft_consensus.cc:729] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166ce3994c93477790f50e73c4849c74, State: Initialized, Role: FOLLOWER
I20260620 02:33:29.931067 19156 consensus_queue.cc:260] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:29.932370 19157 raft_consensus.cc:348] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:29.932847 19157 raft_consensus.cc:374] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:29.932953 19157 raft_consensus.cc:729] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233b9f79fb41468485956acb2401cd71, State: Initialized, Role: FOLLOWER
I20260620 02:33:29.933568 19157 consensus_queue.cc:260] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:29.935305 19156 ts_tablet_manager.cc:1428] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 02:33:29.937964 19158 raft_consensus.cc:348] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:29.938585 19158 raft_consensus.cc:374] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:29.938694 19158 raft_consensus.cc:729] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b08e3d28394638b2a679c07da26eaa, State: Initialized, Role: FOLLOWER
I20260620 02:33:29.938939 19003 heartbeater.cc:503] Master 127.0.97.62:46403 was elected leader, sending a full tablet report...
I20260620 02:33:29.939628 19157 ts_tablet_manager.cc:1428] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260620 02:33:29.939467 19158 consensus_queue.cc:260] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:29.940588 19137 heartbeater.cc:503] Master 127.0.97.62:46403 was elected leader, sending a full tablet report...
I20260620 02:33:29.942479 19158 ts_tablet_manager.cc:1428] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
W20260620 02:33:29.995842 18870 tablet.cc:2369] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:33:30.028301 19004 tablet.cc:2369] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:33:30.055506 19138 tablet.cc:2369] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:30.055794 19164 raft_consensus.cc:482] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:30.056152 19164 raft_consensus.cc:504] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:30.058060 19164 leader_election.cc:288] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166ce3994c93477790f50e73c4849c74 (127.0.97.1:41157), 233b9f79fb41468485956acb2401cd71 (127.0.97.2:40329)
I20260620 02:33:30.065107 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.058692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48339 (local address 127.0.97.2:40329)
0620 02:33:30.058966 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:30.058971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:30.058992 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:33:30.059241 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:30.059249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:30.059341 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:30.059464 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:30.060054 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.060760 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.063086 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.063903 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.064060 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.064159 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:30.064658 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 02:33:30.064778 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:33:30.065115 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.058412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40329 (local address 127.0.97.3:48339)
0620 02:33:30.058820 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:30.058845 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:30.059013 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:30.059484 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:30.059487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:30.059512 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:30.059875 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:30.059888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.060926 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.060931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:30.062918 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:30.062931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.064078 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.064085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:30.064349 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.064377 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:33:30.064604 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 02:33:30.064754 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":69}
I20260620 02:33:30.066329 18958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a858b4efa884533966b41b406ef857a" candidate_uuid: "99b08e3d28394638b2a679c07da26eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233b9f79fb41468485956acb2401cd71" is_pre_election: true
I20260620 02:33:30.066622 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.058567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46533 (local address 127.0.97.1:41157)
0620 02:33:30.058781 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:30.058787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:30.059712 (+ 925us) server_negotiation.cc:407] Connection header received
0620 02:33:30.059803 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:30.059808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:30.059884 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:30.059989 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:30.061522 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.062609 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.064584 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.065734 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.065851 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.065926 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:30.066343 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 02:33:30.066447 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:33:30.066838 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.058456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41157 (local address 127.0.97.3:46533)
0620 02:33:30.059496 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:30.059515 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:30.059661 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:30.060912 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:30.060916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:30.060949 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:30.061333 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:30.061341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.062830 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.062835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:30.064313 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:30.064326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.065944 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.065950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:30.066133 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.066174 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:33:30.066282 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:33:30.066429 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":3,"thread_start_us":546,"threads_started":1}
I20260620 02:33:30.066987 18958 raft_consensus.cc:2454] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99b08e3d28394638b2a679c07da26eaa in term 0.
I20260620 02:33:30.067708 19026 leader_election.cc:302] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233b9f79fb41468485956acb2401cd71, 99b08e3d28394638b2a679c07da26eaa; no voters:
I20260620 02:33:30.067826 18824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a858b4efa884533966b41b406ef857a" candidate_uuid: "99b08e3d28394638b2a679c07da26eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166ce3994c93477790f50e73c4849c74" is_pre_election: true
I20260620 02:33:30.068132 19164 raft_consensus.cc:2781] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:33:30.068287 19164 raft_consensus.cc:482] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:33:30.068416 19164 raft_consensus.cc:3037] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:30.068440 18824 raft_consensus.cc:2454] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99b08e3d28394638b2a679c07da26eaa in term 0.
I20260620 02:33:30.071067 19164 raft_consensus.cc:504] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:30.071751 19164 leader_election.cc:288] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [CANDIDATE]: Term 1 election: Requested vote from peers 166ce3994c93477790f50e73c4849c74 (127.0.97.1:41157), 233b9f79fb41468485956acb2401cd71 (127.0.97.2:40329)
I20260620 02:33:30.072293 18824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a858b4efa884533966b41b406ef857a" candidate_uuid: "99b08e3d28394638b2a679c07da26eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166ce3994c93477790f50e73c4849c74"
I20260620 02:33:30.072393 18958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a858b4efa884533966b41b406ef857a" candidate_uuid: "99b08e3d28394638b2a679c07da26eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233b9f79fb41468485956acb2401cd71"
I20260620 02:33:30.072538 18824 raft_consensus.cc:3037] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:30.072660 18958 raft_consensus.cc:3037] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:30.075416 18824 raft_consensus.cc:2454] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b08e3d28394638b2a679c07da26eaa in term 1.
I20260620 02:33:30.075909 18958 raft_consensus.cc:2454] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b08e3d28394638b2a679c07da26eaa in term 1.
I20260620 02:33:30.075934 19027 leader_election.cc:302] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166ce3994c93477790f50e73c4849c74, 99b08e3d28394638b2a679c07da26eaa; no voters:
I20260620 02:33:30.076357 19164 raft_consensus.cc:2781] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:30.076927 19164 raft_consensus.cc:686] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 1 LEADER]: Becoming Leader. State: Replica: 99b08e3d28394638b2a679c07da26eaa, State: Running, Role: LEADER
I20260620 02:33:30.077450 19164 consensus_queue.cc:237] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } }
I20260620 02:33:30.082230 18679 catalog_manager.cc:5495] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 99b08e3d28394638b2a679c07da26eaa (127.0.97.3). New cstate: current_term: 1 leader_uuid: "99b08e3d28394638b2a679c07da26eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99b08e3d28394638b2a679c07da26eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44407 } health_report { overall_health: HEALTHY } } }
I20260620 02:33:30.136584 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.130234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.97.3:44407)
0620 02:33:30.130548 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:30.130554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:30.130683 (+ 129us) server_negotiation.cc:407] Connection header received
0620 02:33:30.130945 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:30.130950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:30.131034 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:30.131236 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:33:30.131778 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.132609 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.134093 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.134514 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.134664 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.135332 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:33:30.135877 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:33:30.136090 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:30.136259 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 02:33:30.136312 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 02:33:30.145998 18824 raft_consensus.cc:1261] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 1 FOLLOWER]: Refusing update from remote peer 99b08e3d28394638b2a679c07da26eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:30.145965 18958 raft_consensus.cc:1261] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 1 FOLLOWER]: Refusing update from remote peer 99b08e3d28394638b2a679c07da26eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:30.147190 19164 consensus_queue.cc:1035] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "233b9f79fb41468485956acb2401cd71" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:30.147614 19166 consensus_queue.cc:1035] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "166ce3994c93477790f50e73c4849c74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:30.162878 19169 mvcc.cc:205] Tried to move back new op lower bound from 7298755830346465280 to 7298755830081667072. Current Snapshot: MvccSnapshot[applied={T|T < 7298755830346465280}]
I20260620 02:33:30.184486 18680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:30.184798 18680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:30.188820 18680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fcbc624f31e42bc8243c55a8201fcb4: Sending DeleteTablet for 3 replicas of tablet 4a858b4efa884533966b41b406ef857a
I20260620 02:33:30.190279 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a858b4efa884533966b41b406ef857a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36770
I20260620 02:33:30.190313 18938 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a858b4efa884533966b41b406ef857a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34742
I20260620 02:33:30.190295 19072 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a858b4efa884533966b41b406ef857a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47562
I20260620 02:33:30.199122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18741
I20260620 02:33:30.206754 19180 tablet_replica.cc:330] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: stopping tablet replica
I20260620 02:33:30.207444 19180 raft_consensus.cc:2229] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:33:30.208460 19180 raft_consensus.cc:2258] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:30.212030 19180 ts_tablet_manager.cc:1905] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:30.213616 19181 tablet_replica.cc:330] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: stopping tablet replica
I20260620 02:33:30.213907 19179 tablet_replica.cc:330] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: stopping tablet replica
I20260620 02:33:30.214104 19181 raft_consensus.cc:2229] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:30.214372 19179 raft_consensus.cc:2229] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:30.214680 19181 raft_consensus.cc:2258] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:30.214890 19179 raft_consensus.cc:2258] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:30.217823 19181 ts_tablet_manager.cc:1905] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:30.217865 19179 ts_tablet_manager.cc:1905] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:30.221333 19180 ts_tablet_manager.cc:1918] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:30.221573 19180 log.cc:1196] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/4a858b4efa884533966b41b406ef857a
I20260620 02:33:30.222204 19180 ts_tablet_manager.cc:1939] T 4a858b4efa884533966b41b406ef857a P 99b08e3d28394638b2a679c07da26eaa: Deleting consensus metadata
I20260620 02:33:30.224263 18664 catalog_manager.cc:4841] TS 99b08e3d28394638b2a679c07da26eaa (127.0.97.3:44407): tablet 4a858b4efa884533966b41b406ef857a (table test-table [id=58eb28099023462b9d11a132daad8a81]) successfully deleted
I20260620 02:33:30.225386 19179 ts_tablet_manager.cc:1918] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:30.225409 19181 ts_tablet_manager.cc:1918] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:30.225582 19179 log.cc:1196] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/4a858b4efa884533966b41b406ef857a
I20260620 02:33:30.225585 19181 log.cc:1196] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/4a858b4efa884533966b41b406ef857a
I20260620 02:33:30.226174 19179 ts_tablet_manager.cc:1939] T 4a858b4efa884533966b41b406ef857a P 233b9f79fb41468485956acb2401cd71: Deleting consensus metadata
I20260620 02:33:30.226203 19181 ts_tablet_manager.cc:1939] T 4a858b4efa884533966b41b406ef857a P 166ce3994c93477790f50e73c4849c74: Deleting consensus metadata
I20260620 02:33:30.228178 18666 catalog_manager.cc:4841] TS 166ce3994c93477790f50e73c4849c74 (127.0.97.1:41157): tablet 4a858b4efa884533966b41b406ef857a (table test-table [id=58eb28099023462b9d11a132daad8a81]) successfully deleted
I20260620 02:33:30.228343 18665 catalog_manager.cc:4841] TS 233b9f79fb41468485956acb2401cd71 (127.0.97.2:40329): tablet 4a858b4efa884533966b41b406ef857a (table test-table [id=58eb28099023462b9d11a132daad8a81]) successfully deleted
Jun 20 02:33:30 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:33:30.243854 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.200760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.0.97.1:41157)
0620 02:33:30.200981 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:30.200988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:30.215797 (+ 14809us) server_negotiation.cc:407] Connection header received
0620 02:33:30.216035 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:30.216042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:30.216168 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:30.216290 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:30.226691 (+ 10401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.227790 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.229541 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.230230 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.230383 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.235554 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:30.235578 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:30.235600 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:30.235655 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:30.239703 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:30.240869 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:30.240881 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:30.240885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:30.241118 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:30.241654 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:30.241660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:30.241664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:30.242192 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:30.242580 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:30.243137 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 02:33:30.243448 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:33:30.244803 18864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:30.355547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18741
I20260620 02:33:30.375643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18875
Jun 20 02:33:30 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:33:30.395571 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.376604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.0.97.2:40329)
0620 02:33:30.377262 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:30.377267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:30.377285 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:33:30.377345 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:30.377349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:30.377431 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:30.377523 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:30.379309 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.380103 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.382112 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.382535 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.382634 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.387974 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:30.387998 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:30.388021 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:30.388071 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:30.392056 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:30.392911 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:30.392922 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:30.392925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:30.393128 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:30.393630 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:30.393636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:30.393640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:30.394214 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:30.394521 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:30.394917 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 02:33:30.395171 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535}
I20260620 02:33:30.396328 18998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:30.493088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18875
I20260620 02:33:30.512444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19009
Jun 20 02:33:30 dist-test-slave-r706 krb5kdc[18635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:33:30.530545 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:30.513539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.97.3:44407)
0620 02:33:30.513744 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:30.513748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:30.513764 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:33:30.513818 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:30.513822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:30.513922 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:30.514058 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:30.515392 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.516249 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.517768 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:30.518248 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:30.518541 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:30.523631 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:30.523646 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:30.523663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:30.523721 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:30.527304 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:30.528161 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:30.528167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:30.528170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:30.528371 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:30.528793 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:30.528798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:30.528801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:30.529262 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:30.529620 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:30.529920 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 02:33:30.530161 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:33:30.531311 19132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:30.658376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19009
I20260620 02:33:30.679816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 18648
I20260620 02:33:30.681113 18710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:30.795212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 18648
2026-06-20T02:33:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4636 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T02:33:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:30Z Disabled control of system clock
I20260620 02:33:30.882046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39785
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:33:31.235057 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:31.235370 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:31.235458 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:31.242812 19190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:31.242952 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:31.242995 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:31.243047 19190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:31.243094 19190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:31.252856 19190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:31.254251 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:31.256084 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:31.268570 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:31.268748 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:31.269161 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:31.269801 19190 server_base.cc:1034] running on GCE node
I20260620 02:33:31.270623 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:31.272070 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:31.273286 19190 hybrid_clock.cc:648] HybridClock initialized: now 1781922811273242 us; error 29 us; skew 500 ppm
I20260620 02:33:31.273710 19190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:31.276196 19190 webserver.cc:460] Webserver started at http://127.0.97.62:37487/ using document root <none> and password file <none>
I20260620 02:33:31.276923 19190 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:31.277138 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:31.277508 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:31.280561 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e"
format_stamp: "Formatted at 2026-06-20 02:33:31 on dist-test-slave-r706"
server_key: "f59b3b96fb372c47062b7589e79b2e89"
server_key_iv: "f5e62550039d533ea0450e4011158763"
server_key_version: "encryptionkey@0"
I20260620 02:33:31.281421 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e"
format_stamp: "Formatted at 2026-06-20 02:33:31 on dist-test-slave-r706"
server_key: "f59b3b96fb372c47062b7589e79b2e89"
server_key_iv: "f5e62550039d533ea0450e4011158763"
server_key_version: "encryptionkey@0"
I20260620 02:33:31.287735 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260620 02:33:31.292608 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:31.294260 19190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:33:31.294435 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e"
format_stamp: "Formatted at 2026-06-20 02:33:31 on dist-test-slave-r706"
server_key: "f59b3b96fb372c47062b7589e79b2e89"
server_key_iv: "f5e62550039d533ea0450e4011158763"
server_key_version: "encryptionkey@0"
I20260620 02:33:31.294637 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:31.308765 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:31.309830 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:31.310101 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:31.329970 19190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39785
I20260620 02:33:31.329980 19255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39785 every 8 connection(s)
I20260620 02:33:31.331526 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:31.335464 19256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:31.340260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19190
I20260620 02:33:31.340468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:31.341070 388 external_mini_cluster.cc:1428] Setting key dfb111bcd11d066d2c015fa3cdb104a3
I20260620 02:33:31.348232 19256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e: Bootstrap starting.
I20260620 02:33:31.352106 19256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:31.353607 19256 log.cc:824] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:31.357460 19256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e: No bootstrap required, opened a new log
I20260620 02:33:31.357436 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:31.343185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.0.97.62:39785)
0620 02:33:31.344284 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:31.344296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:31.344340 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:33:31.346849 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:31.346876 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:31.347383 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:31.347806 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:31.348433 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:31.350343 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:31.351836 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:31.352829 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:31.353106 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:31.354510 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:31.354517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:31.354533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:31.355196 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:31.355308 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:31.355336 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:31.355476 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:31.355690 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 02:33:31.356058 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":399,"threads_started":1}
I20260620 02:33:31.366313 19256 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39785 } }
I20260620 02:33:31.366822 19256 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:31.366971 19256 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b5f29f5f1642d0a47b8dbdfff18b8e, State: Initialized, Role: FOLLOWER
I20260620 02:33:31.367636 19256 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39785 } }
I20260620 02:33:31.367882 19256 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:31.368029 19256 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:31.368213 19256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:31.370653 19256 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39785 } }
I20260620 02:33:31.371212 19256 leader_election.cc:302] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3b5f29f5f1642d0a47b8dbdfff18b8e; no voters:
I20260620 02:33:31.371878 19256 leader_election.cc:288] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:31.372143 19261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:31.373692 19261 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [term 1 LEADER]: Becoming Leader. State: Replica: e3b5f29f5f1642d0a47b8dbdfff18b8e, State: Running, Role: LEADER
I20260620 02:33:31.374394 19256 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:31.374306 19261 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39785 } }
I20260620 02:33:31.377820 19263 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b5f29f5f1642d0a47b8dbdfff18b8e. Latest consensus state: current_term: 1 leader_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39785 } } }
I20260620 02:33:31.378475 19263 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:31.377775 19262 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b5f29f5f1642d0a47b8dbdfff18b8e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39785 } } }
I20260620 02:33:31.379235 19262 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:31.381417 19271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:31.386811 19271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:31.398042 19271 catalog_manager.cc:1347] Generated new cluster ID: 5c34e0568b964a39b7749711dcb84deb
I20260620 02:33:31.398188 19271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:31.428272 19271 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:31.429700 19271 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:31.440320 19271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e: Generated new TSK 0
I20260620 02:33:31.441365 19271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:31.450141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39785
--builtin_ntp_servers=127.0.97.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:33:31.796173 19280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:31.796625 19280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:31.796788 19280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:31.804939 19280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:31.805116 19280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:31.805281 19280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:31.815227 19280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:31.816521 19280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:31.818372 19280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:31.832261 19285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:31.832293 19286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:31.832417 19288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:31.832935 19280 server_base.cc:1034] running on GCE node
I20260620 02:33:31.833709 19280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:31.834807 19280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:31.836086 19280 hybrid_clock.cc:648] HybridClock initialized: now 1781922811836047 us; error 63 us; skew 500 ppm
I20260620 02:33:31.836455 19280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:31.838891 19280 webserver.cc:460] Webserver started at http://127.0.97.1:35195/ using document root <none> and password file <none>
I20260620 02:33:31.839648 19280 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:31.839785 19280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:31.840118 19280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:31.843103 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "d6749562cfd840c683d0a340d6ea0654"
format_stamp: "Formatted at 2026-06-20 02:33:31 on dist-test-slave-r706"
server_key: "9da9b63ecbbfdcd21cb5c1328f3e6637"
server_key_iv: "bae764a5d8ba31eb645be54221ad0551"
server_key_version: "encryptionkey@0"
I20260620 02:33:31.843912 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d6749562cfd840c683d0a340d6ea0654"
format_stamp: "Formatted at 2026-06-20 02:33:31 on dist-test-slave-r706"
server_key: "9da9b63ecbbfdcd21cb5c1328f3e6637"
server_key_iv: "bae764a5d8ba31eb645be54221ad0551"
server_key_version: "encryptionkey@0"
I20260620 02:33:31.849455 19280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:33:31.853716 19294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:31.854985 19280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:31.855165 19280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "d6749562cfd840c683d0a340d6ea0654"
format_stamp: "Formatted at 2026-06-20 02:33:31 on dist-test-slave-r706"
server_key: "9da9b63ecbbfdcd21cb5c1328f3e6637"
server_key_iv: "bae764a5d8ba31eb645be54221ad0551"
server_key_version: "encryptionkey@0"
I20260620 02:33:31.855397 19280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:31.876075 19280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:31.876911 19280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:31.877238 19280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:31.878238 19280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:31.879952 19280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:31.880081 19280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:31.880204 19280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:31.880293 19280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:31.914572 19280 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39547
I20260620 02:33:31.914662 19406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39547 every 8 connection(s)
I20260620 02:33:31.915895 19280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:31.920905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19280
I20260620 02:33:31.921133 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:33:31.921700 388 external_mini_cluster.cc:1428] Setting key b7839c14e195f6f8369feb18a5144c1d
I20260620 02:33:31.924772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39785
--builtin_ntp_servers=127.0.97.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:33:31.930796 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:31.918572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52089 (local address 127.0.97.62:39785)
0620 02:33:31.919082 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:31.919089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:31.920182 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 02:33:31.922673 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:31.922679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:31.922797 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:31.922929 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:31.924134 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:31.924997 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:31.927636 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:31.928086 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:31.928199 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:31.929909 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:31.929914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:31.929925 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:31.929950 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:31.929998 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:31.930011 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:31.930095 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:31.930397 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 02:33:31.930506 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260620 02:33:31.932288 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:31.919077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39785 (local address 127.0.97.1:52089)
0620 02:33:31.920021 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:31.920063 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:31.922385 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:31.923166 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:31.923174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:31.923265 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:31.923934 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:31.923959 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:31.925189 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:31.925198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:31.927451 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:31.927472 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:31.928337 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:31.928347 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:31.928687 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:31.929524 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:31.929530 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:31.929584 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:31.929702 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:31.930117 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:31.930130 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:31.930142 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:33:31.930344 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 02:33:31.930815 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":319,"threads_started":1}
I20260620 02:33:31.933969 19407 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39785
I20260620 02:33:31.934301 19407 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:31.935060 19407 heartbeater.cc:511] Master 127.0.97.62:39785 requested a full tablet report, sending...
I20260620 02:33:31.937749 19221 ts_manager.cc:194] Registered new tserver with Master: d6749562cfd840c683d0a340d6ea0654 (127.0.97.1:39547)
I20260620 02:33:31.939497 19221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52089
W20260620 02:33:32.270124 19411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:32.270442 19411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:32.270532 19411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:32.278379 19411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:32.278498 19411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:32.278687 19411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:32.288025 19411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:32.289490 19411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:32.291281 19411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:32.306228 19416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:32.306130 19417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:32.306258 19419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:32.307106 19411 server_base.cc:1034] running on GCE node
I20260620 02:33:32.307796 19411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:32.308866 19411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:32.310119 19411 hybrid_clock.cc:648] HybridClock initialized: now 1781922812310069 us; error 65 us; skew 500 ppm
I20260620 02:33:32.310472 19411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:32.312790 19411 webserver.cc:460] Webserver started at http://127.0.97.2:44949/ using document root <none> and password file <none>
I20260620 02:33:32.313561 19411 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:32.313701 19411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:32.314050 19411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:32.316987 19411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "607f2099a2754e189084aeb091da8b78"
format_stamp: "Formatted at 2026-06-20 02:33:32 on dist-test-slave-r706"
server_key: "1e819ed81bc5922c19b589cb5074e825"
server_key_iv: "d768b596c083f8b676762cac2b14e3b6"
server_key_version: "encryptionkey@0"
I20260620 02:33:32.317873 19411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "607f2099a2754e189084aeb091da8b78"
format_stamp: "Formatted at 2026-06-20 02:33:32 on dist-test-slave-r706"
server_key: "1e819ed81bc5922c19b589cb5074e825"
server_key_iv: "d768b596c083f8b676762cac2b14e3b6"
server_key_version: "encryptionkey@0"
I20260620 02:33:32.323294 19411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:33:32.327026 19425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:32.328377 19411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:33:32.328557 19411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "607f2099a2754e189084aeb091da8b78"
format_stamp: "Formatted at 2026-06-20 02:33:32 on dist-test-slave-r706"
server_key: "1e819ed81bc5922c19b589cb5074e825"
server_key_iv: "d768b596c083f8b676762cac2b14e3b6"
server_key_version: "encryptionkey@0"
I20260620 02:33:32.328804 19411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:32.365517 19411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:32.366370 19411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:32.366633 19411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:32.367621 19411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:32.369244 19411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:32.369338 19411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:32.369426 19411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:32.369482 19411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:32.404093 19411 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40455
I20260620 02:33:32.404184 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40455 every 8 connection(s)
I20260620 02:33:32.405488 19411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:32.407055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19411
I20260620 02:33:32.407262 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:33:32.407733 388 external_mini_cluster.cc:1428] Setting key 34abb4f231efb806339fa3e17a5ec20f
I20260620 02:33:32.410808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39785
--builtin_ntp_servers=127.0.97.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:33:32.418308 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.408103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38341 (local address 127.0.97.62:39785)
0620 02:33:32.408324 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:32.408331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:32.409901 (+ 1570us) server_negotiation.cc:407] Connection header received
0620 02:33:32.411783 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:32.411789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:32.411873 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:32.411999 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:32.413076 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.413792 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.415360 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.415763 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.415899 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.417289 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:32.417295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:32.417300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:32.417326 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:32.417357 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:32.417368 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:32.417467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:32.417799 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 02:33:32.417916 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:33:32.419090 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.408652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39785 (local address 127.0.97.2:38341)
0620 02:33:32.409715 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:32.409762 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:32.411525 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:32.412226 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:32.412234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:32.412320 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:32.412899 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.412915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.413941 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.413946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.415204 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.415216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.415983 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.415990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.416207 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.416940 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:32.416943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:32.416979 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:32.417108 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:32.417465 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:32.417477 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:32.417488 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:33:32.417711 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 02:33:32.418002 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":392,"threads_started":1}
I20260620 02:33:32.420598 19538 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39785
I20260620 02:33:32.420899 19538 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:32.421666 19538 heartbeater.cc:511] Master 127.0.97.62:39785 requested a full tablet report, sending...
I20260620 02:33:32.423210 19221 ts_manager.cc:194] Registered new tserver with Master: 607f2099a2754e189084aeb091da8b78 (127.0.97.2:40455)
I20260620 02:33:32.424053 19221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38341
W20260620 02:33:32.753139 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:32.753507 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:32.753672 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:32.761443 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:32.761597 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:32.761787 19541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:32.771731 19541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:32.773346 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:32.775903 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:32.796376 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:32.796398 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:32.796754 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:32.797781 19541 server_base.cc:1034] running on GCE node
I20260620 02:33:32.798666 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:32.800182 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:32.801429 19541 hybrid_clock.cc:648] HybridClock initialized: now 1781922812801375 us; error 49 us; skew 500 ppm
I20260620 02:33:32.801842 19541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:32.804888 19541 webserver.cc:460] Webserver started at http://127.0.97.3:46749/ using document root <none> and password file <none>
I20260620 02:33:32.805979 19541 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:32.806118 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:32.806525 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:32.810611 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "0720497043c741898727805e6fb05395"
format_stamp: "Formatted at 2026-06-20 02:33:32 on dist-test-slave-r706"
server_key: "b8a84eb7cfb02455e8c9cb0fb1d85814"
server_key_iv: "fd584749b83687e49efdf9a63eee3a42"
server_key_version: "encryptionkey@0"
I20260620 02:33:32.811738 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0720497043c741898727805e6fb05395"
format_stamp: "Formatted at 2026-06-20 02:33:32 on dist-test-slave-r706"
server_key: "b8a84eb7cfb02455e8c9cb0fb1d85814"
server_key_iv: "fd584749b83687e49efdf9a63eee3a42"
server_key_version: "encryptionkey@0"
I20260620 02:33:32.819576 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 02:33:32.823794 19555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:32.825745 19541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:33:32.825937 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "0720497043c741898727805e6fb05395"
format_stamp: "Formatted at 2026-06-20 02:33:32 on dist-test-slave-r706"
server_key: "b8a84eb7cfb02455e8c9cb0fb1d85814"
server_key_iv: "fd584749b83687e49efdf9a63eee3a42"
server_key_version: "encryptionkey@0"
I20260620 02:33:32.826260 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:32.841188 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:32.842039 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:32.842332 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:32.843350 19541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:32.844885 19541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:32.845009 19541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:32.845182 19541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:32.845270 19541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:32.881233 19541 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40481
I20260620 02:33:32.881283 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40481 every 8 connection(s)
I20260620 02:33:32.882580 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:32.891122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19541
I20260620 02:33:32.891352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:33:32.891824 388 external_mini_cluster.cc:1428] Setting key 9282649de59a0e7fc2e3e1259bf2723e
I20260620 02:33:32.895411 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.885160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47337 (local address 127.0.97.62:39785)
0620 02:33:32.885345 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:32.885350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:32.886667 (+ 1317us) server_negotiation.cc:407] Connection header received
0620 02:33:32.888470 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:32.888476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:32.888569 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:32.888681 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:32.889805 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.890573 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.892341 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.892861 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.892981 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.894566 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:32.894573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:32.894578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:32.894599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:32.894641 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:32.894657 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:32.894755 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:32.895061 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:33:32.895150 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:33:32.896436 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.885636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39785 (local address 127.0.97.3:47337)
0620 02:33:32.886497 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:32.886539 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:32.888198 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:32.888918 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:32.888926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:32.889060 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:32.889633 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.889650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.890735 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.890739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.892167 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.892179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.893146 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.893157 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.893422 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.894222 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:32.894225 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:32.894267 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:32.894348 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:32.894789 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:32.894801 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:32.894809 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:33:32.894982 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 02:33:32.895295 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":279,"threads_started":1}
I20260620 02:33:32.898102 19668 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39785
I20260620 02:33:32.898489 19668 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:32.899288 19668 heartbeater.cc:511] Master 127.0.97.62:39785 requested a full tablet report, sending...
I20260620 02:33:32.900985 19221 ts_manager.cc:194] Registered new tserver with Master: 0720497043c741898727805e6fb05395 (127.0.97.3:40481)
I20260620 02:33:32.901839 19221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47337
I20260620 02:33:32.907256 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:33:32.918354 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.911702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.0.97.62:39785)
0620 02:33:32.911994 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:32.912000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:32.912300 (+ 300us) server_negotiation.cc:407] Connection header received
0620 02:33:32.912438 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:32.912442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:32.912522 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:32.912620 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:32.913228 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.914163 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.915560 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.916202 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.916333 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.917532 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:32.917539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:32.917544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:32.917570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:32.917613 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:32.917630 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:32.917798 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:32.917999 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 02:33:32.918057 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:33:32.922745 19221 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:33:32.929852 19221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:33:32.943794 19407 heartbeater.cc:503] Master 127.0.97.62:39785 was elected leader, sending a full tablet report...
I20260620 02:33:32.954241 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.943185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40455 (local address 127.0.0.1:47214)
0620 02:33:32.944592 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:32.944644 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:32.944928 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:32.946066 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:32.946075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:32.946115 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:32.946614 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.946633 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.948717 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.948722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.950833 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.950848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.953067 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.953076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.953348 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.953426 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 02:33:32.953575 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 02:33:32.953827 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":941,"threads_started":1}
I20260620 02:33:32.955839 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.943399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.0.97.2:40455)
0620 02:33:32.943978 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:32.943988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:32.944749 (+ 761us) server_negotiation.cc:407] Connection header received
0620 02:33:32.945292 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:32.945305 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:32.945626 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:32.945815 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:32.946864 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.948475 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.951058 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.952494 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.952628 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.955099 (+ 2471us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:32.955191 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 02:33:32.955404 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":835,"threads_started":1}
I20260620 02:33:32.955979 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.945559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.97.3:40481)
0620 02:33:32.946524 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:32.946534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:32.947074 (+ 540us) server_negotiation.cc:407] Connection header received
0620 02:33:32.947398 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:32.947407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:32.947607 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:32.947751 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:32.948726 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.949859 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.952747 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.954259 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.954378 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.954677 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:32.955322 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 02:33:32.955514 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":604,"threads_started":1}
I20260620 02:33:32.956779 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.944936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40481 (local address 127.0.0.1:56156)
0620 02:33:32.946969 (+ 2033us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:32.946988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:32.947180 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:32.948048 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:32.948053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:32.948076 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:32.948521 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.948537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.950015 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.950018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.952462 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.952484 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.954891 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.954898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.955090 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.955123 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:33:32.956245 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0620 02:33:32.956399 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1905,"mutex_wait_us":22,"thread_start_us":1758,"threads_started":1}
I20260620 02:33:32.958140 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.944058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.0.97.1:39547)
0620 02:33:32.944830 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:32.944839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:32.946357 (+ 1518us) server_negotiation.cc:407] Connection header received
0620 02:33:32.946477 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:32.946489 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:32.946743 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:32.946924 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:32.948477 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.950077 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.952884 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.955362 (+ 2478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.956897 (+ 1535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.957445 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:32.957525 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 02:33:32.957686 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":1366,"threads_started":1}
I20260620 02:33:32.958896 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:32.943622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41442)
0620 02:33:32.946043 (+ 2421us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:32.946066 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:32.946210 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:32.947186 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:32.947191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:32.947212 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:32.948242 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.948256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.950227 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.950232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.952548 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:32.952557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:32.955621 (+ 3064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:32.955631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:32.955951 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:32.956307 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 02:33:32.956442 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 02:33:32.956619 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2247,"mutex_wait_us":28,"thread_start_us":2109,"threads_started":1}
I20260620 02:33:32.961393 19603 tablet_service.cc:1467] Processing CreateTablet for tablet 174f75daf9d448b2a0dd582cee183674 (DEFAULT_TABLE table=test-table [id=ec74af0b27d243e4b6b78fe7e01a5a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:32.963092 19603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174f75daf9d448b2a0dd582cee183674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:32.963083 19342 tablet_service.cc:1467] Processing CreateTablet for tablet 174f75daf9d448b2a0dd582cee183674 (DEFAULT_TABLE table=test-table [id=ec74af0b27d243e4b6b78fe7e01a5a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:32.963361 19473 tablet_service.cc:1467] Processing CreateTablet for tablet 174f75daf9d448b2a0dd582cee183674 (DEFAULT_TABLE table=test-table [id=ec74af0b27d243e4b6b78fe7e01a5a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:32.964540 19342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174f75daf9d448b2a0dd582cee183674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:32.964665 19473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174f75daf9d448b2a0dd582cee183674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:32.976152 19686 tablet_bootstrap.cc:492] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Bootstrap starting.
I20260620 02:33:32.978327 19687 tablet_bootstrap.cc:492] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Bootstrap starting.
I20260620 02:33:32.979955 19688 tablet_bootstrap.cc:492] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Bootstrap starting.
I20260620 02:33:32.982157 19686 tablet_bootstrap.cc:654] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:32.982471 19687 tablet_bootstrap.cc:654] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:32.982847 19688 tablet_bootstrap.cc:654] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:32.983543 19686 log.cc:824] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:32.983618 19687 log.cc:824] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:32.983899 19688 log.cc:824] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:32.987210 19686 tablet_bootstrap.cc:492] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: No bootstrap required, opened a new log
I20260620 02:33:32.987227 19687 tablet_bootstrap.cc:492] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: No bootstrap required, opened a new log
I20260620 02:33:32.987542 19687 ts_tablet_manager.cc:1397] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:33:32.987540 19686 ts_tablet_manager.cc:1397] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260620 02:33:32.988039 19688 tablet_bootstrap.cc:492] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: No bootstrap required, opened a new log
I20260620 02:33:32.988308 19688 ts_tablet_manager.cc:1397] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 02:33:32.995527 19688 raft_consensus.cc:348] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:32.995988 19688 raft_consensus.cc:374] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:32.996088 19688 raft_consensus.cc:729] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607f2099a2754e189084aeb091da8b78, State: Initialized, Role: FOLLOWER
I20260620 02:33:32.996762 19688 consensus_queue.cc:260] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:32.998019 19538 heartbeater.cc:503] Master 127.0.97.62:39785 was elected leader, sending a full tablet report...
I20260620 02:33:32.998837 19688 ts_tablet_manager.cc:1428] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 02:33:32.998684 19686 raft_consensus.cc:348] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:32.998693 19687 raft_consensus.cc:348] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:32.999140 19686 raft_consensus.cc:374] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:32.999140 19687 raft_consensus.cc:374] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:32.999234 19687 raft_consensus.cc:729] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0720497043c741898727805e6fb05395, State: Initialized, Role: FOLLOWER
I20260620 02:33:32.999251 19686 raft_consensus.cc:729] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6749562cfd840c683d0a340d6ea0654, State: Initialized, Role: FOLLOWER
I20260620 02:33:32.999941 19687 consensus_queue.cc:260] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:32.999945 19686 consensus_queue.cc:260] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:33.001911 19687 ts_tablet_manager.cc:1428] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260620 02:33:33.002537 19668 heartbeater.cc:503] Master 127.0.97.62:39785 was elected leader, sending a full tablet report...
I20260620 02:33:33.002764 19686 ts_tablet_manager.cc:1428] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
W20260620 02:33:33.134462 19669 tablet.cc:2369] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:33:33.157786 19539 tablet.cc:2369] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:33:33.168578 19408 tablet.cc:2369] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:33.170943 19694 raft_consensus.cc:482] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:33.171298 19694 raft_consensus.cc:504] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:33.173144 19694 leader_election.cc:288] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607f2099a2754e189084aeb091da8b78 (127.0.97.2:40455), 0720497043c741898727805e6fb05395 (127.0.97.3:40481)
I20260620 02:33:33.181995 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.173693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53519 (local address 127.0.97.2:40455)
0620 02:33:33.173968 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:33.173976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:33.174438 (+ 462us) server_negotiation.cc:407] Connection header received
0620 02:33:33.174773 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:33.174782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:33.174905 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:33.175022 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:33.175944 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.176928 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.179642 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.180876 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.181054 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.181161 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:33.181614 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 02:33:33.181733 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:33:33.182142 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.173608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40455 (local address 127.0.97.1:53519)
0620 02:33:33.174290 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.174311 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:33.174565 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:33.175074 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:33.175079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:33.175116 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:33.175700 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:33.175716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.177173 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.177177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:33.179375 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:33.179392 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.181245 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.181249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:33.181371 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.181445 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 02:33:33.181563 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 02:33:33.181712 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":84,"thread_start_us":401,"threads_started":1}
I20260620 02:33:33.182287 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.174748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58231 (local address 127.0.97.3:40481)
0620 02:33:33.174961 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:33.174968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:33.175416 (+ 448us) server_negotiation.cc:407] Connection header received
0620 02:33:33.175835 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:33.175842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:33.175934 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:33.176050 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:33.176578 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.177700 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.180076 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.181358 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.181464 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.181573 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:33.181913 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 02:33:33.182052 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:33:33.182926 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.174565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40481 (local address 127.0.97.1:58231)
0620 02:33:33.175308 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.175333 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:33.175646 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:33.176114 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:33.176117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:33.176137 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:33.176401 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:33.176408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.177858 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.177864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:33.179876 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:33.179886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.181565 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.181573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:33.181741 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.181771 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:33:33.181860 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:33:33.181979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":6,"thread_start_us":401,"threads_started":1}
I20260620 02:33:33.183485 19493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75daf9d448b2a0dd582cee183674" candidate_uuid: "d6749562cfd840c683d0a340d6ea0654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607f2099a2754e189084aeb091da8b78" is_pre_election: true
I20260620 02:33:33.184079 19493 raft_consensus.cc:2454] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6749562cfd840c683d0a340d6ea0654 in term 0.
I20260620 02:33:33.184104 19623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75daf9d448b2a0dd582cee183674" candidate_uuid: "d6749562cfd840c683d0a340d6ea0654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0720497043c741898727805e6fb05395" is_pre_election: true
I20260620 02:33:33.184705 19623 raft_consensus.cc:2454] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6749562cfd840c683d0a340d6ea0654 in term 0.
I20260620 02:33:33.184893 19295 leader_election.cc:302] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f2099a2754e189084aeb091da8b78, d6749562cfd840c683d0a340d6ea0654; no voters:
I20260620 02:33:33.185441 19694 raft_consensus.cc:2781] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:33:33.185586 19694 raft_consensus.cc:482] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:33:33.185729 19694 raft_consensus.cc:3037] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:33.188179 19694 raft_consensus.cc:504] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:33.188931 19694 leader_election.cc:288] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [CANDIDATE]: Term 1 election: Requested vote from peers 607f2099a2754e189084aeb091da8b78 (127.0.97.2:40455), 0720497043c741898727805e6fb05395 (127.0.97.3:40481)
I20260620 02:33:33.190056 19493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75daf9d448b2a0dd582cee183674" candidate_uuid: "d6749562cfd840c683d0a340d6ea0654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607f2099a2754e189084aeb091da8b78"
I20260620 02:33:33.190056 19623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174f75daf9d448b2a0dd582cee183674" candidate_uuid: "d6749562cfd840c683d0a340d6ea0654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0720497043c741898727805e6fb05395"
I20260620 02:33:33.190300 19493 raft_consensus.cc:3037] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:33.190300 19623 raft_consensus.cc:3037] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:33.193794 19623 raft_consensus.cc:2454] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6749562cfd840c683d0a340d6ea0654 in term 1.
I20260620 02:33:33.193794 19493 raft_consensus.cc:2454] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6749562cfd840c683d0a340d6ea0654 in term 1.
I20260620 02:33:33.194336 19295 leader_election.cc:302] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f2099a2754e189084aeb091da8b78, d6749562cfd840c683d0a340d6ea0654; no voters:
I20260620 02:33:33.194708 19694 raft_consensus.cc:2781] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:33.195233 19694 raft_consensus.cc:686] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 1 LEADER]: Becoming Leader. State: Replica: d6749562cfd840c683d0a340d6ea0654, State: Running, Role: LEADER
I20260620 02:33:33.195748 19694 consensus_queue.cc:237] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } }
I20260620 02:33:33.201512 19219 catalog_manager.cc:5495] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6749562cfd840c683d0a340d6ea0654 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d6749562cfd840c683d0a340d6ea0654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6749562cfd840c683d0a340d6ea0654" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39547 } health_report { overall_health: HEALTHY } } }
I20260620 02:33:33.261333 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.255180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.97.1:39547)
0620 02:33:33.255510 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:33.255517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:33.255633 (+ 116us) server_negotiation.cc:407] Connection header received
0620 02:33:33.255894 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:33.255898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:33.255993 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:33.256164 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:33:33.256564 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.257423 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.258892 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.259336 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.259488 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.260143 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:33:33.260664 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:33:33.260880 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:33.261028 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 02:33:33.261075 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 02:33:33.270656 19493 raft_consensus.cc:1261] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 1 FOLLOWER]: Refusing update from remote peer d6749562cfd840c683d0a340d6ea0654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:33.270669 19623 raft_consensus.cc:1261] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 1 FOLLOWER]: Refusing update from remote peer d6749562cfd840c683d0a340d6ea0654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:33.271857 19697 consensus_queue.cc:1035] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607f2099a2754e189084aeb091da8b78" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:33.272284 19694 consensus_queue.cc:1035] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0720497043c741898727805e6fb05395" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:33.300925 19219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:33.301354 19219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:33.305125 19219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3b5f29f5f1642d0a47b8dbdfff18b8e: Sending DeleteTablet for 3 replicas of tablet 174f75daf9d448b2a0dd582cee183674
I20260620 02:33:33.306525 19473 tablet_service.cc:1514] Processing DeleteTablet for tablet 174f75daf9d448b2a0dd582cee183674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:33 UTC) from {username='slave'} at 127.0.0.1:47214
I20260620 02:33:33.306563 19342 tablet_service.cc:1514] Processing DeleteTablet for tablet 174f75daf9d448b2a0dd582cee183674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:33 UTC) from {username='slave'} at 127.0.0.1:41442
I20260620 02:33:33.306828 19603 tablet_service.cc:1514] Processing DeleteTablet for tablet 174f75daf9d448b2a0dd582cee183674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:33 UTC) from {username='slave'} at 127.0.0.1:56156
I20260620 02:33:33.308928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19280
I20260620 02:33:33.319237 19712 tablet_replica.cc:330] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: stopping tablet replica
I20260620 02:33:33.319955 19712 raft_consensus.cc:2229] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:33.320545 19712 raft_consensus.cc:2258] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:33.323155 19712 ts_tablet_manager.cc:1905] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:33.327905 19710 tablet_replica.cc:330] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: stopping tablet replica
I20260620 02:33:33.328428 19710 raft_consensus.cc:2229] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:33:33.329237 19710 raft_consensus.cc:2258] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:33.331908 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.310328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.0.97.1:39547)
0620 02:33:33.310506 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:33.310512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:33.316846 (+ 6334us) server_negotiation.cc:407] Connection header received
0620 02:33:33.317927 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:33.317936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:33.318079 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:33.318174 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:33.319527 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.320743 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.324427 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.325181 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.325295 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.329260 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:33.329273 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:33.329291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:33.330578 (+ 1287us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:33.330660 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:33.330695 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:33.330878 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:33.331214 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 02:33:33.331371 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:33:33.333549 19402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:33.333760 19711 tablet_replica.cc:330] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: stopping tablet replica
I20260620 02:33:33.334501 19711 raft_consensus.cc:2229] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:33.335317 19711 raft_consensus.cc:2258] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:33.337916 19711 ts_tablet_manager.cc:1905] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:33.339557 19712 ts_tablet_manager.cc:1918] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:33.339876 19712 log.cc:1196] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/174f75daf9d448b2a0dd582cee183674
I20260620 02:33:33.340482 19712 ts_tablet_manager.cc:1939] T 174f75daf9d448b2a0dd582cee183674 P 0720497043c741898727805e6fb05395: Deleting consensus metadata
I20260620 02:33:33.342756 19205 catalog_manager.cc:4841] TS 0720497043c741898727805e6fb05395 (127.0.97.3:40481): tablet 174f75daf9d448b2a0dd582cee183674 (table test-table [id=ec74af0b27d243e4b6b78fe7e01a5a65]) successfully deleted
I20260620 02:33:33.345386 19711 ts_tablet_manager.cc:1918] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:33.345543 19711 log.cc:1196] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/174f75daf9d448b2a0dd582cee183674
I20260620 02:33:33.346071 19711 ts_tablet_manager.cc:1939] T 174f75daf9d448b2a0dd582cee183674 P 607f2099a2754e189084aeb091da8b78: Deleting consensus metadata
I20260620 02:33:33.347817 19205 catalog_manager.cc:4841] TS 607f2099a2754e189084aeb091da8b78 (127.0.97.2:40455): tablet 174f75daf9d448b2a0dd582cee183674 (table test-table [id=ec74af0b27d243e4b6b78fe7e01a5a65]) successfully deleted
I20260620 02:33:33.334661 19710 ts_tablet_manager.cc:1905] T 174f75daf9d448b2a0dd582cee183674 P d6749562cfd840c683d0a340d6ea0654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:33.446612 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19280
I20260620 02:33:33.466838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19411
W20260620 02:33:33.466799 19207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39547 (error 108)
I20260620 02:33:33.468895 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.468095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41472)
0620 02:33:33.468536 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.468680 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":207,"threads_started":1}
W20260620 02:33:33.469177 19207 catalog_manager.cc:4584] TS d6749562cfd840c683d0a340d6ea0654 (127.0.97.1:39547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 174f75daf9d448b2a0dd582cee183674: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
I20260620 02:33:33.474586 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.467664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.0.97.2:40455)
0620 02:33:33.468080 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:33.468086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:33.468106 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:33:33.468179 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:33.468187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:33.468349 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:33.468488 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:33.468963 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.469929 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.471222 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.471634 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.471737 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.472862 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:33.472871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:33.472885 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:33.473672 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:33.473727 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:33.473748 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:33.473863 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:33.474030 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 02:33:33.474229 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260620 02:33:33.475435 19533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:33.523207 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.522626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41488)
0620 02:33:33.522919 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.523027 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 02:33:33.524345 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.523992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41502)
0620 02:33:33.524142 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.524213 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:33:33.569161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19411
I20260620 02:33:33.585083 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.584394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41516)
0620 02:33:33.584679 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.584815 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 02:33:33.586447 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.585902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41528)
0620 02:33:33.586098 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.586200 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:33:33.587809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19541
I20260620 02:33:33.595141 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.588706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.0.97.3:40481)
0620 02:33:33.588893 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:33.588901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:33.588960 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:33:33.589228 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:33.589233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:33.589336 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:33.589457 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:33.589900 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.590616 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.591929 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:33.592336 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:33.592499 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:33.593692 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:33.593702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:33.593724 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:33.594293 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:33.594342 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:33.594361 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:33.594495 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:33.594686 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 02:33:33.594812 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:33:33.595887 19663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:33.667295 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.666767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41544)
0620 02:33:33.667019 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.667139 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 02:33:33.668469 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:33.668027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39547 (local address 127.0.0.1:41554)
0620 02:33:33.668250 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:33.668323 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 02:33:33.686143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19541
I20260620 02:33:33.703459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19190
I20260620 02:33:33.704543 19251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:33.815618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19190
2026-06-20T02:33:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3019 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:33 dist-test-slave-r706 krb5kdc[19722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:33 dist-test-slave-r706 krb5kdc[19722](info): set up 2 sockets
Jun 20 02:33:33 dist-test-slave-r706 krb5kdc[19722](info): commencing operation
krb5kdc: starting...
W20260620 02:33:34.953426 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.005s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:35 dist-test-slave-r706 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:33:35.246637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:35.595750 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:35.596072 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:35.596237 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:35.603951 19735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:35.604080 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:35.604166 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:35.604255 19735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:35.604350 19735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:35.614758 19735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:35.616164 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:35.617558 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:35.630092 19740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:35.630056 19741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:35.630056 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:35.630874 19735 server_base.cc:1034] running on GCE node
I20260620 02:33:35.631747 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:35.633239 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:35.634465 19735 hybrid_clock.cc:648] HybridClock initialized: now 1781922815634402 us; error 55 us; skew 500 ppm
I20260620 02:33:35.634809 19735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:35 dist-test-slave-r706 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:35.637970 19735 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:33:35.639461 19735 webserver.cc:460] Webserver started at http://127.0.97.62:36919/ using document root <none> and password file <none>
I20260620 02:33:35.640187 19735 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:35.640340 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:35.640678 19735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:35.643805 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "06e61b0f2f6445eebbc9ecfacece10a8"
format_stamp: "Formatted at 2026-06-20 02:33:35 on dist-test-slave-r706"
server_key: "cd3eb92702e5e439973ae0791cd0058d"
server_key_iv: "1b7090a7f5f40cf33e2cc06cfadab260"
server_key_version: "encryptionkey@0"
I20260620 02:33:35.644675 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "06e61b0f2f6445eebbc9ecfacece10a8"
format_stamp: "Formatted at 2026-06-20 02:33:35 on dist-test-slave-r706"
server_key: "cd3eb92702e5e439973ae0791cd0058d"
server_key_iv: "1b7090a7f5f40cf33e2cc06cfadab260"
server_key_version: "encryptionkey@0"
I20260620 02:33:35.650112 19735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:33:35.653975 19750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:35.655375 19735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:35.655579 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "06e61b0f2f6445eebbc9ecfacece10a8"
format_stamp: "Formatted at 2026-06-20 02:33:35 on dist-test-slave-r706"
server_key: "cd3eb92702e5e439973ae0791cd0058d"
server_key_iv: "1b7090a7f5f40cf33e2cc06cfadab260"
server_key_version: "encryptionkey@0"
I20260620 02:33:35.655809 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:35.680519 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:35.684885 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:35.685207 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:35.703991 19735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40509
I20260620 02:33:35.704028 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40509 every 8 connection(s)
I20260620 02:33:35.705546 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:35.709442 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:35.715662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19735
I20260620 02:33:35.715902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:35.716365 388 external_mini_cluster.cc:1428] Setting key e714930d28cfce13bd10ca5336fa2fa7
I20260620 02:33:35.721431 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8: Bootstrap starting.
I20260620 02:33:35.725431 19802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:35.726843 19802 log.cc:824] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:35.730687 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8: No bootstrap required, opened a new log
Jun 20 02:33:35 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:35.741825 19802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40509 } }
I20260620 02:33:35.742316 19802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:35.742442 19802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e61b0f2f6445eebbc9ecfacece10a8, State: Initialized, Role: FOLLOWER
I20260620 02:33:35.743158 19802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40509 } }
I20260620 02:33:35.743427 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:35.718448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.0.97.62:40509)
0620 02:33:35.719390 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:35.719402 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:35.719446 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:33:35.721296 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:35.721322 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:35.721763 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:35.722100 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:35.724097 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:35.725553 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:35.727533 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:35.728643 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:35.728790 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:35.733913 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:35.733925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:35.733946 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:35.734000 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:35.738294 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:35.739108 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:35.739118 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:35.739127 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:35.739389 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:35.739754 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:35.739760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:35.739765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:35.740569 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:35.740880 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:35.741409 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 02:33:35.741879 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":464,"threads_started":1}
I20260620 02:33:35.743484 19802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:35.743623 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:35.743837 19802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:35.746358 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40509 } }
I20260620 02:33:35.746862 19802 leader_election.cc:302] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e61b0f2f6445eebbc9ecfacece10a8; no voters:
I20260620 02:33:35.747476 19802 leader_election.cc:288] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:35.747648 19807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:35.748435 19807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [term 1 LEADER]: Becoming Leader. State: Replica: 06e61b0f2f6445eebbc9ecfacece10a8, State: Running, Role: LEADER
I20260620 02:33:35.749200 19807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40509 } }
I20260620 02:33:35.749962 19802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:35.752112 19808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40509 } } }
I20260620 02:33:35.752486 19808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:35.753427 19812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:35.753252 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e61b0f2f6445eebbc9ecfacece10a8. Latest consensus state: current_term: 1 leader_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e61b0f2f6445eebbc9ecfacece10a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40509 } } }
I20260620 02:33:35.753608 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:35.760629 19812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:35.770135 19812 catalog_manager.cc:1347] Generated new cluster ID: fb722764c36b4e5eb3911baf772eff1f
I20260620 02:33:35.770303 19812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:35.796310 19812 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:35.797761 19812 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:35.808465 19812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8: Generated new TSK 0
I20260620 02:33:35.809547 19812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:33:35.884106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40509
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:36.235680 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:36.236001 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:36.236163 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:36.243808 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:36.243932 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:36.244131 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:36.254798 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:36.256177 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:36.257769 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:36.272516 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:36.272472 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:36.272544 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:36.273798 19828 server_base.cc:1034] running on GCE node
I20260620 02:33:36.274535 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:36.275674 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:36.276949 19828 hybrid_clock.cc:648] HybridClock initialized: now 1781922816276903 us; error 63 us; skew 500 ppm
I20260620 02:33:36.277407 19828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:36 dist-test-slave-r706 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:36.281121 19828 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:33:36.282801 19828 webserver.cc:460] Webserver started at http://127.0.97.1:42125/ using document root <none> and password file <none>
I20260620 02:33:36.283519 19828 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:36.283638 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:36.283933 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:36.287194 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "52b81e4ddc4b46fba037683200dfd360"
format_stamp: "Formatted at 2026-06-20 02:33:36 on dist-test-slave-r706"
server_key: "0bebbcab4118d07fa027c43bad472753"
server_key_iv: "02e6b0dbebd9dee5b0dd1b7471c9e0e1"
server_key_version: "encryptionkey@0"
I20260620 02:33:36.288082 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "52b81e4ddc4b46fba037683200dfd360"
format_stamp: "Formatted at 2026-06-20 02:33:36 on dist-test-slave-r706"
server_key: "0bebbcab4118d07fa027c43bad472753"
server_key_iv: "02e6b0dbebd9dee5b0dd1b7471c9e0e1"
server_key_version: "encryptionkey@0"
I20260620 02:33:36.294077 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:33:36.298061 19843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:36.299571 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:36.299790 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "52b81e4ddc4b46fba037683200dfd360"
format_stamp: "Formatted at 2026-06-20 02:33:36 on dist-test-slave-r706"
server_key: "0bebbcab4118d07fa027c43bad472753"
server_key_iv: "02e6b0dbebd9dee5b0dd1b7471c9e0e1"
server_key_version: "encryptionkey@0"
I20260620 02:33:36.300007 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:36.343451 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:36.348075 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:36.348409 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:36.349459 19828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:36.350970 19828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:36.351045 19828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:36.351173 19828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:36.351238 19828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:36.385821 19828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40605
I20260620 02:33:36.385914 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40605 every 8 connection(s)
I20260620 02:33:36.387125 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:36.394364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19828
I20260620 02:33:36.394558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:33:36.395018 388 external_mini_cluster.cc:1428] Setting key 21c196816b32fa558a0dee11876d0d79
Jun 20 02:33:36 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:36.408795 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:36.389772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53099 (local address 127.0.97.62:40509)
0620 02:33:36.390201 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:36.390205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:36.391140 (+ 935us) server_negotiation.cc:407] Connection header received
0620 02:33:36.393048 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:36.393054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:36.393148 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:36.393268 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:36.394816 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.395587 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.397460 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.397860 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.397994 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:36.402018 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:36.402026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:36.402033 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:36.402074 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:36.405268 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:36.406214 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:36.406218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:36.406221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:36.406332 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:36.406822 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:36.406825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:36.406827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:36.407078 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:36.407261 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:36.408241 (+ 980us) server_negotiation.cc:299] Negotiation successful
0620 02:33:36.408485 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":199,"threads_started":1}
I20260620 02:33:36.409844 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:36.390307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40509 (local address 127.0.97.1:53099)
0620 02:33:36.390982 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:36.391020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:36.392749 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:36.393492 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:36.393500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:36.394005 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:36.394644 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:36.394673 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.395741 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.395746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:36.397320 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:36.397333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.398091 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.398098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:36.398343 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:36.399012 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:36.399036 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:36.401791 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:36.405402 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:36.405414 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:36.405434 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:36.406089 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:36.406433 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:36.406440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:36.406447 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:36.406710 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:36.407343 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:36.407354 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:36.407811 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 02:33:36.408143 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 02:33:36.408681 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260620 02:33:36.411448 19956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40509
I20260620 02:33:36.411777 19956 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:36.412490 19956 heartbeater.cc:511] Master 127.0.97.62:40509 requested a full tablet report, sending...
I20260620 02:33:36.414578 19767 ts_manager.cc:194] Registered new tserver with Master: 52b81e4ddc4b46fba037683200dfd360 (127.0.97.1:40605)
I20260620 02:33:36.416015 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53099
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:33:36.461069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40509
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:36.794538 19962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:36.794835 19962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:36.794998 19962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:36.802359 19962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:36.802471 19962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:36.802675 19962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:36.813097 19962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:36.814370 19962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:36.815656 19962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:36.829694 19970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:36.829995 19967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:36.830032 19968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:36.830736 19962 server_base.cc:1034] running on GCE node
I20260620 02:33:36.831490 19962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:36.832549 19962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:36.833791 19962 hybrid_clock.cc:648] HybridClock initialized: now 1781922816833749 us; error 58 us; skew 500 ppm
I20260620 02:33:36.834124 19962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:36 dist-test-slave-r706 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:36.837450 19962 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:33:36.838975 19962 webserver.cc:460] Webserver started at http://127.0.97.2:43383/ using document root <none> and password file <none>
I20260620 02:33:36.839679 19962 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:36.839816 19962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:36.840127 19962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:36.843086 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "9dad2f7afeaa477eb83572215b0b0ba1"
format_stamp: "Formatted at 2026-06-20 02:33:36 on dist-test-slave-r706"
server_key: "b23177c9ab94a08d3edb31fb9b0c4c53"
server_key_iv: "f7a0817eec6e112f9b463c81008f204d"
server_key_version: "encryptionkey@0"
I20260620 02:33:36.843889 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9dad2f7afeaa477eb83572215b0b0ba1"
format_stamp: "Formatted at 2026-06-20 02:33:36 on dist-test-slave-r706"
server_key: "b23177c9ab94a08d3edb31fb9b0c4c53"
server_key_iv: "f7a0817eec6e112f9b463c81008f204d"
server_key_version: "encryptionkey@0"
I20260620 02:33:36.849133 19962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:33:36.853149 19977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:36.854638 19962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:33:36.854825 19962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "9dad2f7afeaa477eb83572215b0b0ba1"
format_stamp: "Formatted at 2026-06-20 02:33:36 on dist-test-slave-r706"
server_key: "b23177c9ab94a08d3edb31fb9b0c4c53"
server_key_iv: "f7a0817eec6e112f9b463c81008f204d"
server_key_version: "encryptionkey@0"
I20260620 02:33:36.855059 19962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:36.876015 19962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:36.880674 19962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:36.880987 19962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:36.882061 19962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:36.883497 19962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:36.883584 19962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:36.883719 19962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:36.883780 19962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:36.917217 19962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39223
I20260620 02:33:36.917325 20089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39223 every 8 connection(s)
I20260620 02:33:36.918496 19962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:36.920035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 19962
I20260620 02:33:36.920229 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:33:36.920761 388 external_mini_cluster.cc:1428] Setting key 981b5de381be8aa714f11bd1b1266679
Jun 20 02:33:36 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:36.941174 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:36.921214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55729 (local address 127.0.97.62:40509)
0620 02:33:36.921719 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:36.921725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:36.922690 (+ 965us) server_negotiation.cc:407] Connection header received
0620 02:33:36.924588 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:36.924594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:36.924687 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:36.924804 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:36.926382 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.927137 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.928693 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.929193 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.929328 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:36.933175 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:36.933186 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:36.933192 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:36.933245 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:36.937275 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:36.938098 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:36.938107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:36.938114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:36.938296 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:36.938775 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:36.938783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:36.938790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:36.939236 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:36.939430 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:36.940340 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 02:33:36.940628 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":243,"threads_started":1}
I20260620 02:33:36.941946 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:36.921742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40509 (local address 127.0.97.2:55729)
0620 02:33:36.922481 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:36.922524 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:36.924320 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:36.925098 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:36.925107 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:36.925636 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:36.926199 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:36.926217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.927285 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.927295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:36.928530 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:36.928541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:36.929424 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:36.929432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:36.929673 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:36.930389 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:36.930417 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:36.932878 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:36.937459 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:36.937467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:36.937482 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:36.937957 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:36.938431 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:36.938434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:36.938437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:36.938641 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:36.939423 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:36.939430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:36.939921 (+ 491us) client_negotiation.cc:769] Sending connection context
0620 02:33:36.940209 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 02:33:36.940642 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":311,"threads_started":1}
I20260620 02:33:36.943528 20090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40509
I20260620 02:33:36.943912 20090 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:36.944835 20090 heartbeater.cc:511] Master 127.0.97.62:40509 requested a full tablet report, sending...
I20260620 02:33:36.946571 19767 ts_manager.cc:194] Registered new tserver with Master: 9dad2f7afeaa477eb83572215b0b0ba1 (127.0.97.2:39223)
I20260620 02:33:36.947453 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55729
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:33:36.990201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40509
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:37.330019 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:37.330317 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:37.330413 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:37.337792 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:37.337905 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:37.338019 20096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:37.348165 20096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:37.349493 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:37.351300 20096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:37.365140 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:37.365093 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:37.365180 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:37.366608 20096 server_base.cc:1034] running on GCE node
I20260620 02:33:37.367285 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:37.368292 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:37.369496 20096 hybrid_clock.cc:648] HybridClock initialized: now 1781922817369446 us; error 46 us; skew 500 ppm
I20260620 02:33:37.369838 20096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:37 dist-test-slave-r706 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:37.373164 20096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:33:37.374631 20096 webserver.cc:460] Webserver started at http://127.0.97.3:43063/ using document root <none> and password file <none>
I20260620 02:33:37.375379 20096 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:37.375509 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:37.375849 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:37.378942 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "acbec2b82b2b46e6ab8026dc6337a4a5"
format_stamp: "Formatted at 2026-06-20 02:33:37 on dist-test-slave-r706"
server_key: "50517431d8d42b762409a15a38e9f132"
server_key_iv: "420d321f1bde4863623258e816ecf1ea"
server_key_version: "encryptionkey@0"
I20260620 02:33:37.379796 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "acbec2b82b2b46e6ab8026dc6337a4a5"
format_stamp: "Formatted at 2026-06-20 02:33:37 on dist-test-slave-r706"
server_key: "50517431d8d42b762409a15a38e9f132"
server_key_iv: "420d321f1bde4863623258e816ecf1ea"
server_key_version: "encryptionkey@0"
I20260620 02:33:37.384980 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:33:37.388700 20111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:37.390115 20096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:33:37.390285 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "acbec2b82b2b46e6ab8026dc6337a4a5"
format_stamp: "Formatted at 2026-06-20 02:33:37 on dist-test-slave-r706"
server_key: "50517431d8d42b762409a15a38e9f132"
server_key_iv: "420d321f1bde4863623258e816ecf1ea"
server_key_version: "encryptionkey@0"
I20260620 02:33:37.390489 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:37.418836 19956 heartbeater.cc:503] Master 127.0.97.62:40509 was elected leader, sending a full tablet report...
I20260620 02:33:37.419569 20096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:37.423982 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:37.424283 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:37.425298 20096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:37.426694 20096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:37.426802 20096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:37.426944 20096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:37.427012 20096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:37.461201 20096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43961
I20260620 02:33:37.461265 20223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43961 every 8 connection(s)
I20260620 02:33:37.462555 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:37.469424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20096
I20260620 02:33:37.469633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:33:37.470067 388 external_mini_cluster.cc:1428] Setting key 7a7b5e1bf2fe015c0e238b7012c3db18
Jun 20 02:33:37 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:33:37.484747 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.465303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43399 (local address 127.0.97.62:40509)
0620 02:33:37.465805 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.465810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.466714 (+ 904us) server_negotiation.cc:407] Connection header received
0620 02:33:37.468740 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.468748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.468843 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.468970 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:37.470814 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.471688 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.473406 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.473821 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.473963 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.477936 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:37.477942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:37.477945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:37.477981 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:37.481435 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.482271 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.482275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.482278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.482401 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.482862 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.482866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.482868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.483156 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:37.483311 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.484087 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.484333 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
I20260620 02:33:37.485915 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.465790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40509 (local address 127.0.97.3:43399)
0620 02:33:37.466501 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:37.466545 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:37.468394 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:37.469270 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:37.469279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:37.469936 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:37.470611 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.470631 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.471865 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.471875 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.473255 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.473269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.474062 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.474069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.474327 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.475072 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:33:37.475097 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:37.477645 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:37.481612 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:37.481623 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:37.481647 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:37.482124 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:37.482520 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:33:37.482525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:33:37.482529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:33:37.482729 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:33:37.483300 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:37.483309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:37.483669 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 02:33:37.483938 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 02:33:37.484560 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":288,"threads_started":1}
I20260620 02:33:37.487290 20224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40509
I20260620 02:33:37.487668 20224 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:37.488379 20224 heartbeater.cc:511] Master 127.0.97.62:40509 requested a full tablet report, sending...
I20260620 02:33:37.489934 19767 ts_manager.cc:194] Registered new tserver with Master: acbec2b82b2b46e6ab8026dc6337a4a5 (127.0.97.3:43961)
I20260620 02:33:37.490824 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43399
I20260620 02:33:37.498128 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:33:37.515491 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.502460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.97.62:40509)
0620 02:33:37.502636 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.502640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.503124 (+ 484us) server_negotiation.cc:407] Connection header received
0620 02:33:37.503259 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.503264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.503334 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.503427 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:37.504962 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.505859 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.507152 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.507590 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.507700 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.510156 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:37.510162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:37.510165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:37.510216 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:37.513114 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.513784 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.513788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.513790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.513919 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.514228 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.514231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.514233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.514563 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:37.514759 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.515075 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.515255 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:33:37.519556 19767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:33:37.526665 19767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:33:37.549824 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.539066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:39408)
0620 02:33:37.540531 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:37.540575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:37.540729 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:37.543012 (+ 2283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:37.543019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:37.543051 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:37.543461 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.543474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.545427 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.545434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.547522 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.547534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.548822 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.548828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.549041 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.549110 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 02:33:37.549254 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 02:33:37.549439 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":1056,"threads_started":1}
I20260620 02:33:37.550719 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.539409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.97.1:40605)
0620 02:33:37.542313 (+ 2904us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.542324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.542358 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:33:37.542479 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.542488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.542755 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.543692 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:37.543940 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.545304 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.547688 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.548617 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.548771 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.549046 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.549291 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.549454 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2731,"thread_start_us":2573,"threads_started":1}
I20260620 02:33:37.551643 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.539203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43961 (local address 127.0.0.1:39034)
0620 02:33:37.541880 (+ 2677us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:37.541899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:37.543243 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:37.543970 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:37.543974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:37.544000 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:37.544451 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.544464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.546384 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.546391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.548666 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.548676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.550800 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.550806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.550991 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.551026 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:33:37.551096 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 02:33:37.551196 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2545,"mutex_wait_us":4,"thread_start_us":2392,"threads_started":1}
I20260620 02:33:37.551780 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.542323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.97.2:39223)
0620 02:33:37.543180 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.543188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.543214 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:33:37.543313 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.543322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.543544 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.543727 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:37.544374 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.545955 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.548837 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.550309 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.550434 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.551274 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.551353 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.551505 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":574,"threads_started":1}
I20260620 02:33:37.552508 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.539879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.0.97.3:43961)
0620 02:33:37.541397 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.541407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.543152 (+ 1745us) server_negotiation.cc:407] Connection header received
0620 02:33:37.543448 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.543454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.543605 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.543797 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:37.544629 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.546194 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.548837 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.550309 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.550434 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.550666 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.551882 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.552154 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":1155,"threads_started":1}
I20260620 02:33:37.551676 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.541759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39223 (local address 127.0.0.1:33164)
0620 02:33:37.542562 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:37.542582 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:37.542696 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:37.543742 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:37.543749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:37.543772 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:37.544199 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.544210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.546103 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.546109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.548626 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.548646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.550698 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.550706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.550904 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.550937 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:33:37.551017 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:33:37.551166 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":92,"thread_start_us":713,"threads_started":1}
I20260620 02:33:37.555218 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 4d95bfcca7db4960882119532487473f (DEFAULT_TABLE table=test-table [id=9f7eaffae7e7411fbb56c3304f6358c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:37.555727 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 4d95bfcca7db4960882119532487473f (DEFAULT_TABLE table=test-table [id=9f7eaffae7e7411fbb56c3304f6358c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:37.556066 20159 tablet_service.cc:1467] Processing CreateTablet for tablet 4d95bfcca7db4960882119532487473f (DEFAULT_TABLE table=test-table [id=9f7eaffae7e7411fbb56c3304f6358c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:37.556874 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d95bfcca7db4960882119532487473f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:37.556980 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d95bfcca7db4960882119532487473f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:37.557360 20159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d95bfcca7db4960882119532487473f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:37.570140 20243 tablet_bootstrap.cc:492] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Bootstrap starting.
I20260620 02:33:37.573603 20243 tablet_bootstrap.cc:654] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:37.573817 20245 tablet_bootstrap.cc:492] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Bootstrap starting.
I20260620 02:33:37.575171 20243 log.cc:824] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:37.575672 20244 tablet_bootstrap.cc:492] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Bootstrap starting.
I20260620 02:33:37.577881 20245 tablet_bootstrap.cc:654] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:37.579154 20245 log.cc:824] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:37.579341 20244 tablet_bootstrap.cc:654] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:37.579370 20243 tablet_bootstrap.cc:492] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: No bootstrap required, opened a new log
I20260620 02:33:37.579654 20243 ts_tablet_manager.cc:1397] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:33:37.580540 20244 log.cc:824] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:37.582086 20245 tablet_bootstrap.cc:492] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: No bootstrap required, opened a new log
I20260620 02:33:37.582335 20245 ts_tablet_manager.cc:1397] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 02:33:37.583482 20244 tablet_bootstrap.cc:492] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: No bootstrap required, opened a new log
I20260620 02:33:37.583750 20244 ts_tablet_manager.cc:1397] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 02:33:37.589706 20245 raft_consensus.cc:348] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.590111 20245 raft_consensus.cc:374] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:37.590226 20245 raft_consensus.cc:729] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acbec2b82b2b46e6ab8026dc6337a4a5, State: Initialized, Role: FOLLOWER
I20260620 02:33:37.590207 20243 raft_consensus.cc:348] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.590693 20243 raft_consensus.cc:374] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:37.590833 20243 raft_consensus.cc:729] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dad2f7afeaa477eb83572215b0b0ba1, State: Initialized, Role: FOLLOWER
I20260620 02:33:37.590939 20245 consensus_queue.cc:260] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.591548 20243 consensus_queue.cc:260] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.592150 20224 heartbeater.cc:503] Master 127.0.97.62:40509 was elected leader, sending a full tablet report...
I20260620 02:33:37.592844 20245 ts_tablet_manager.cc:1428] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 02:33:37.593780 20090 heartbeater.cc:503] Master 127.0.97.62:40509 was elected leader, sending a full tablet report...
I20260620 02:33:37.594365 20243 ts_tablet_manager.cc:1428] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 02:33:37.595844 20244 raft_consensus.cc:348] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.596339 20244 raft_consensus.cc:374] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:37.596468 20244 raft_consensus.cc:729] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b81e4ddc4b46fba037683200dfd360, State: Initialized, Role: FOLLOWER
I20260620 02:33:37.597169 20244 consensus_queue.cc:260] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.598950 20244 ts_tablet_manager.cc:1428] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260620 02:33:37.639647 19957 tablet.cc:2369] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:37.656898 20250 raft_consensus.cc:482] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:37.657339 20250 raft_consensus.cc:504] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.659091 20250 leader_election.cc:288] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52b81e4ddc4b46fba037683200dfd360 (127.0.97.1:40605), acbec2b82b2b46e6ab8026dc6337a4a5 (127.0.97.3:43961)
I20260620 02:33:37.666790 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.659605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44095 (local address 127.0.97.1:40605)
0620 02:33:37.659856 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.659862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.660944 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 02:33:37.661138 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.661147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.661241 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.661345 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:37.662156 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.663216 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.665080 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.665839 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.666002 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.666125 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.666395 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.666524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:33:37.666970 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.659973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.97.2:44095)
0620 02:33:37.660808 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:37.660825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:37.660957 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:37.661453 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:37.661463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:37.661496 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:37.661941 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.661955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.663364 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.663368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.664855 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.664867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.666071 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.666077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.666236 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.666260 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:33:37.666374 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:33:37.666512 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"spinlock_wait_cycles":5760,"thread_start_us":489,"threads_started":1}
I20260620 02:33:37.667161 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.659668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50645 (local address 127.0.97.3:43961)
0620 02:33:37.659889 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.659895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.661579 (+ 1684us) server_negotiation.cc:407] Connection header received
0620 02:33:37.661881 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.661886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.661968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.662051 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:37.662784 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.663705 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.665450 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.666210 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.666358 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.666465 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.666760 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.666897 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:33:37.667321 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.660001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43961 (local address 127.0.97.2:50645)
0620 02:33:37.661474 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:37.661501 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:37.661705 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:37.662171 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:37.662178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:37.662203 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:37.662600 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.662617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.663848 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.663854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.665300 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:37.665312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.666422 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.666428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:37.666598 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.666621 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:33:37.666723 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:33:37.666845 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":1178,"threads_started":1}
I20260620 02:33:37.668234 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d95bfcca7db4960882119532487473f" candidate_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b81e4ddc4b46fba037683200dfd360" is_pre_election: true
I20260620 02:33:37.668507 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d95bfcca7db4960882119532487473f" candidate_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" is_pre_election: true
I20260620 02:33:37.668756 19911 raft_consensus.cc:2454] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dad2f7afeaa477eb83572215b0b0ba1 in term 0.
I20260620 02:33:37.669039 20179 raft_consensus.cc:2454] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dad2f7afeaa477eb83572215b0b0ba1 in term 0.
I20260620 02:33:37.669528 19980 leader_election.cc:302] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b81e4ddc4b46fba037683200dfd360, 9dad2f7afeaa477eb83572215b0b0ba1; no voters:
I20260620 02:33:37.669904 20250 raft_consensus.cc:2781] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:33:37.670048 20250 raft_consensus.cc:482] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:33:37.670166 20250 raft_consensus.cc:3037] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:33:37.670608 20091 tablet.cc:2369] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:37.672820 20250 raft_consensus.cc:504] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.673523 20250 leader_election.cc:288] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [CANDIDATE]: Term 1 election: Requested vote from peers 52b81e4ddc4b46fba037683200dfd360 (127.0.97.1:40605), acbec2b82b2b46e6ab8026dc6337a4a5 (127.0.97.3:43961)
I20260620 02:33:37.673919 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d95bfcca7db4960882119532487473f" candidate_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b81e4ddc4b46fba037683200dfd360"
I20260620 02:33:37.673971 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d95bfcca7db4960882119532487473f" candidate_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5"
I20260620 02:33:37.674163 20179 raft_consensus.cc:3037] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:37.674152 19911 raft_consensus.cc:3037] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:37.677454 20179 raft_consensus.cc:2454] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dad2f7afeaa477eb83572215b0b0ba1 in term 1.
I20260620 02:33:37.677449 19911 raft_consensus.cc:2454] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dad2f7afeaa477eb83572215b0b0ba1 in term 1.
I20260620 02:33:37.677881 19981 leader_election.cc:302] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dad2f7afeaa477eb83572215b0b0ba1, acbec2b82b2b46e6ab8026dc6337a4a5; no voters:
I20260620 02:33:37.678211 20250 raft_consensus.cc:2781] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:37.678648 20250 raft_consensus.cc:686] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 9dad2f7afeaa477eb83572215b0b0ba1, State: Running, Role: LEADER
I20260620 02:33:37.679124 20250 consensus_queue.cc:237] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } }
I20260620 02:33:37.683921 19767 catalog_manager.cc:5495] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dad2f7afeaa477eb83572215b0b0ba1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dad2f7afeaa477eb83572215b0b0ba1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39223 } health_report { overall_health: HEALTHY } } }
I20260620 02:33:37.704753 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.698623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.97.2:39223)
0620 02:33:37.698808 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.698812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.698837 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:33:37.698927 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.698930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.699009 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.699152 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:33:37.699510 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.700353 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.702385 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.702807 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.702939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.703414 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:33:37.703954 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:33:37.704112 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.704314 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.704400 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:33:37.713739 20179 raft_consensus.cc:1261] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 1 FOLLOWER]: Refusing update from remote peer 9dad2f7afeaa477eb83572215b0b0ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:37.713739 19911 raft_consensus.cc:1261] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 1 FOLLOWER]: Refusing update from remote peer 9dad2f7afeaa477eb83572215b0b0ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 02:33:37.714293 20225 tablet.cc:2369] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:33:37.714831 20254 consensus_queue.cc:1035] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b81e4ddc4b46fba037683200dfd360" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:37.715212 20250 consensus_queue.cc:1035] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acbec2b82b2b46e6ab8026dc6337a4a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:37.726331 20257 mvcc.cc:205] Tried to move back new op lower bound from 7298755861344133120 to 7298755861217681408. Current Snapshot: MvccSnapshot[applied={T|T < 7298755861344133120}]
I20260620 02:33:37.744751 19767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:37.745074 19767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:37.748826 19767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e61b0f2f6445eebbc9ecfacece10a8: Sending DeleteTablet for 3 replicas of tablet 4d95bfcca7db4960882119532487473f
I20260620 02:33:37.750101 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d95bfcca7db4960882119532487473f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39408
I20260620 02:33:37.750566 20159 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d95bfcca7db4960882119532487473f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39034
I20260620 02:33:37.751226 20025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d95bfcca7db4960882119532487473f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33164
I20260620 02:33:37.751816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19828
I20260620 02:33:37.762137 20267 tablet_replica.cc:330] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: stopping tablet replica
I20260620 02:33:37.762449 20269 tablet_replica.cc:330] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: stopping tablet replica
I20260620 02:33:37.762688 20267 raft_consensus.cc:2229] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:37.763053 20269 raft_consensus.cc:2229] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:33:37.763291 20267 raft_consensus.cc:2258] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:37.763888 20269 raft_consensus.cc:2258] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:37.764219 20268 tablet_replica.cc:330] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: stopping tablet replica
I20260620 02:33:37.764657 20268 raft_consensus.cc:2229] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:37.765223 20268 raft_consensus.cc:2258] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:37.766461 20269 ts_tablet_manager.cc:1905] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:37.767298 20267 ts_tablet_manager.cc:1905] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:37.767545 20268 ts_tablet_manager.cc:1905] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:37.774485 20269 ts_tablet_manager.cc:1918] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:37.774811 20269 log.cc:1196] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/4d95bfcca7db4960882119532487473f
I20260620 02:33:37.774864 20267 ts_tablet_manager.cc:1918] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:37.775170 20267 log.cc:1196] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-0/wal/wals/4d95bfcca7db4960882119532487473f
I20260620 02:33:37.775543 20269 ts_tablet_manager.cc:1939] T 4d95bfcca7db4960882119532487473f P 9dad2f7afeaa477eb83572215b0b0ba1: Deleting consensus metadata
I20260620 02:33:37.775828 20267 ts_tablet_manager.cc:1939] T 4d95bfcca7db4960882119532487473f P 52b81e4ddc4b46fba037683200dfd360: Deleting consensus metadata
I20260620 02:33:37.777880 19753 catalog_manager.cc:4841] TS 52b81e4ddc4b46fba037683200dfd360 (127.0.97.1:40605): tablet 4d95bfcca7db4960882119532487473f (table test-table [id=9f7eaffae7e7411fbb56c3304f6358c4]) successfully deleted
I20260620 02:33:37.778230 19752 catalog_manager.cc:4841] TS 9dad2f7afeaa477eb83572215b0b0ba1 (127.0.97.2:39223): tablet 4d95bfcca7db4960882119532487473f (table test-table [id=9f7eaffae7e7411fbb56c3304f6358c4]) successfully deleted
I20260620 02:33:37.779471 20268 ts_tablet_manager.cc:1918] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:37.779656 20268 log.cc:1196] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/4d95bfcca7db4960882119532487473f
I20260620 02:33:37.780100 20268 ts_tablet_manager.cc:1939] T 4d95bfcca7db4960882119532487473f P acbec2b82b2b46e6ab8026dc6337a4a5: Deleting consensus metadata
I20260620 02:33:37.781703 19754 catalog_manager.cc:4841] TS acbec2b82b2b46e6ab8026dc6337a4a5 (127.0.97.3:43961): tablet 4d95bfcca7db4960882119532487473f (table test-table [id=9f7eaffae7e7411fbb56c3304f6358c4]) successfully deleted
Jun 20 02:33:37 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:33:37.790956 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.752977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.0.97.1:40605)
0620 02:33:37.753236 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.753246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.760260 (+ 7014us) server_negotiation.cc:407] Connection header received
0620 02:33:37.760684 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.760695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.760846 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.760942 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:37.767775 (+ 6833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.769064 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.775199 (+ 6135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.775947 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.776056 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.785072 (+ 9016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:37.785090 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:37.785114 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:37.785157 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:37.787985 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.788739 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.788747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.788749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.788887 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.789320 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.789325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.789328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.789697 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:37.790024 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.790335 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.790567 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":97}
I20260620 02:33:37.791652 19951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:37.884537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19828
I20260620 02:33:37.903982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19962
Jun 20 02:33:37 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:33:37.922747 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:37.904915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.0.97.2:39223)
0620 02:33:37.905151 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:37.905157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:37.905183 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:33:37.905336 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:37.905342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:37.905427 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:37.905552 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:37.906840 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.908072 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.909497 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:37.910205 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:37.910336 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:37.914837 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:37.914856 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:37.914882 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:37.914929 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:37.919430 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.920258 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.920268 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.920272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.920443 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:37.920852 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:37.920857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:37.920861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:37.921365 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:37.921724 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:37.922099 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:33:37.922337 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 02:33:37.923568 20085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:38.040277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19962
I20260620 02:33:38.060695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20096
Jun 20 02:33:38 dist-test-slave-r706 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:33:38.077987 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:38.061491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.0.97.3:43961)
0620 02:33:38.061705 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:38.061709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:38.061729 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:33:38.061891 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:38.061896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:38.061969 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:38.062062 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:38.063632 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:38.064478 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:38.066105 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:38.066538 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:38.066648 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:38.071724 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:38.071742 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:38.071757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:38.071803 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:38.074922 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:38.075616 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:38.075623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:38.075626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:38.075776 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:38.076243 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:38.076247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:38.076250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:38.076699 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:38.076945 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:38.077376 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 02:33:38.077606 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:33:38.078662 20219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:38.169687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20096
I20260620 02:33:38.187180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 19735
I20260620 02:33:38.188414 19797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:38.298938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 19735
2026-06-20T02:33:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4484 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T02:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:38Z Disabled control of system clock
I20260620 02:33:38.387053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37413
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:33:38.728550 20278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:38.728875 20278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:38.728960 20278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:38.736537 20278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:38.736655 20278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:38.736693 20278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:38.736754 20278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:38.736799 20278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:38.746275 20278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:38.747598 20278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:38.749418 20278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:38.761310 20283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:38.761368 20286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:38.761245 20284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:38.762387 20278 server_base.cc:1034] running on GCE node
I20260620 02:33:38.763283 20278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:38.764695 20278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:38.765928 20278 hybrid_clock.cc:648] HybridClock initialized: now 1781922818765884 us; error 56 us; skew 500 ppm
I20260620 02:33:38.766302 20278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:38.768635 20278 webserver.cc:460] Webserver started at http://127.0.97.62:36233/ using document root <none> and password file <none>
I20260620 02:33:38.769394 20278 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:38.769529 20278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:38.769850 20278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:38.772961 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "782b948feb55493b88b240afe2d71853"
format_stamp: "Formatted at 2026-06-20 02:33:38 on dist-test-slave-r706"
server_key: "8bb2c8e4ae38223969b4df87c2d0262a"
server_key_iv: "d9b9c0ddef6ea047ccb62fe3e6abf52d"
server_key_version: "encryptionkey@0"
I20260620 02:33:38.773846 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "782b948feb55493b88b240afe2d71853"
format_stamp: "Formatted at 2026-06-20 02:33:38 on dist-test-slave-r706"
server_key: "8bb2c8e4ae38223969b4df87c2d0262a"
server_key_iv: "d9b9c0ddef6ea047ccb62fe3e6abf52d"
server_key_version: "encryptionkey@0"
I20260620 02:33:38.779150 20278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:33:38.782776 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:38.784126 20278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:33:38.784335 20278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "782b948feb55493b88b240afe2d71853"
format_stamp: "Formatted at 2026-06-20 02:33:38 on dist-test-slave-r706"
server_key: "8bb2c8e4ae38223969b4df87c2d0262a"
server_key_iv: "d9b9c0ddef6ea047ccb62fe3e6abf52d"
server_key_version: "encryptionkey@0"
I20260620 02:33:38.784565 20278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:38.799821 20278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:38.800676 20278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:38.800971 20278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:38.820606 20278 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37413
I20260620 02:33:38.820632 20343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37413 every 8 connection(s)
I20260620 02:33:38.822074 20278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:38.824126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20278
I20260620 02:33:38.824308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:38.824753 388 external_mini_cluster.cc:1428] Setting key a198e2ce84120813439ef5ade8fa0c00
I20260620 02:33:38.826370 20344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:38.838821 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:38.826741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.0.97.62:37413)
0620 02:33:38.827734 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:38.827745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:38.827785 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:33:38.829446 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:38.829469 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:38.829919 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:38.830261 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:38.830901 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:38.832324 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:38.834257 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:38.834918 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:38.835060 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:38.836330 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:38.836338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:38.836350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:38.836955 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:38.837102 (+ 147us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:38.837127 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:38.837224 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:38.837499 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 02:33:38.837846 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":384,"threads_started":1}
I20260620 02:33:38.840058 20344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853: Bootstrap starting.
I20260620 02:33:38.843201 20344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:38.844417 20344 log.cc:824] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:38.847321 20344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853: No bootstrap required, opened a new log
I20260620 02:33:38.854920 20344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782b948feb55493b88b240afe2d71853" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37413 } }
I20260620 02:33:38.855377 20344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:38.855506 20344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782b948feb55493b88b240afe2d71853, State: Initialized, Role: FOLLOWER
I20260620 02:33:38.856033 20344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782b948feb55493b88b240afe2d71853" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37413 } }
I20260620 02:33:38.856237 20344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:38.856333 20344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:38.856464 20344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:38.858731 20344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782b948feb55493b88b240afe2d71853" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37413 } }
I20260620 02:33:38.859277 20344 leader_election.cc:302] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782b948feb55493b88b240afe2d71853; no voters:
I20260620 02:33:38.859874 20344 leader_election.cc:288] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:38.860097 20349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:38.860846 20349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [term 1 LEADER]: Becoming Leader. State: Replica: 782b948feb55493b88b240afe2d71853, State: Running, Role: LEADER
I20260620 02:33:38.861435 20349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782b948feb55493b88b240afe2d71853" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37413 } }
I20260620 02:33:38.862213 20344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:38.864594 20350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782b948feb55493b88b240afe2d71853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782b948feb55493b88b240afe2d71853" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37413 } } }
I20260620 02:33:38.864626 20351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782b948feb55493b88b240afe2d71853. Latest consensus state: current_term: 1 leader_uuid: "782b948feb55493b88b240afe2d71853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782b948feb55493b88b240afe2d71853" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37413 } } }
I20260620 02:33:38.865063 20350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:38.865144 20351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:38.866930 20356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:38.872164 20356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:38.881505 20356 catalog_manager.cc:1347] Generated new cluster ID: 2d80395f552d463f828c6ef6c555131a
I20260620 02:33:38.881644 20356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:38.905772 20356 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:38.907094 20356 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:38.918035 20356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853: Generated new TSK 0
I20260620 02:33:38.919142 20356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:38.935007 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37413
--builtin_ntp_servers=127.0.97.20:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:33:39.280437 20368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:39.280777 20368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:39.280943 20368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:39.288455 20368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:39.288561 20368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:39.288766 20368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:33:39.298327 20368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:39.299611 20368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:39.301749 20368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:39.314954 20374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:39.314924 20373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:39.315076 20376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:39.316119 20368 server_base.cc:1034] running on GCE node
I20260620 02:33:39.316752 20368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:39.317831 20368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:39.319077 20368 hybrid_clock.cc:648] HybridClock initialized: now 1781922819319022 us; error 41 us; skew 500 ppm
I20260620 02:33:39.319464 20368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:39.325584 20368 webserver.cc:460] Webserver started at http://127.0.97.1:33417/ using document root <none> and password file <none>
I20260620 02:33:39.326314 20368 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:39.326412 20368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:39.326702 20368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:39.329770 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data/instance:
uuid: "497c2ef0904d42eba65d161d46269a1a"
format_stamp: "Formatted at 2026-06-20 02:33:39 on dist-test-slave-r706"
server_key: "575d2ddaf12fd929b66640a30cb82464"
server_key_iv: "4c177c1b9d6102e9c95817f64efef4e0"
server_key_version: "encryptionkey@0"
I20260620 02:33:39.330518 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal/instance:
uuid: "497c2ef0904d42eba65d161d46269a1a"
format_stamp: "Formatted at 2026-06-20 02:33:39 on dist-test-slave-r706"
server_key: "575d2ddaf12fd929b66640a30cb82464"
server_key_iv: "4c177c1b9d6102e9c95817f64efef4e0"
server_key_version: "encryptionkey@0"
I20260620 02:33:39.335660 20368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:33:39.339345 20382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:39.340799 20368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:33:39.340971 20368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal
uuid: "497c2ef0904d42eba65d161d46269a1a"
format_stamp: "Formatted at 2026-06-20 02:33:39 on dist-test-slave-r706"
server_key: "575d2ddaf12fd929b66640a30cb82464"
server_key_iv: "4c177c1b9d6102e9c95817f64efef4e0"
server_key_version: "encryptionkey@0"
I20260620 02:33:39.341274 20368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:39.358165 20368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:39.358985 20368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:39.359290 20368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:39.360289 20368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:39.361915 20368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:39.362053 20368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:39.362164 20368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:39.362255 20368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:39.396708 20368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42663
I20260620 02:33:39.397032 20494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42663 every 8 connection(s)
I20260620 02:33:39.398202 20368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:33:39.398978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20368
I20260620 02:33:39.399147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:33:39.399734 388 external_mini_cluster.cc:1428] Setting key 7d7707f0db05f3039c4c6a8926920e4e
I20260620 02:33:39.402715 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37413
--builtin_ntp_servers=127.0.97.20:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:33:39.412313 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:39.401262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59567 (local address 127.0.97.62:37413)
0620 02:33:39.401764 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:39.401771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:39.403157 (+ 1386us) server_negotiation.cc:407] Connection header received
0620 02:33:39.404914 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:39.404923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:39.405045 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:39.405173 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:39.406289 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.407428 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.409051 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.409773 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.409913 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:39.411303 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:39.411312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:39.411325 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:39.411351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:39.411404 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:39.411422 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:39.411530 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:39.411736 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 02:33:39.411849 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
I20260620 02:33:39.413153 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:39.401931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37413 (local address 127.0.97.1:59567)
0620 02:33:39.402985 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:39.403035 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:39.404650 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:39.405403 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:39.405412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:39.405501 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:39.406108 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:39.406126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.407590 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.407594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:39.408865 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:39.408881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.409999 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.410004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:39.410217 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:39.410969 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:39.410971 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:39.411013 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:39.411105 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:39.411525 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:39.411534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:39.411540 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:33:39.411696 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:33:39.411996 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":367,"threads_started":1}
I20260620 02:33:39.414762 20495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37413
I20260620 02:33:39.415114 20495 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:39.416038 20495 heartbeater.cc:511] Master 127.0.97.62:37413 requested a full tablet report, sending...
I20260620 02:33:39.418447 20309 ts_manager.cc:194] Registered new tserver with Master: 497c2ef0904d42eba65d161d46269a1a (127.0.97.1:42663)
I20260620 02:33:39.419934 20309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59567
W20260620 02:33:39.743572 20499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:39.743923 20499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:39.744104 20499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:39.751525 20499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:39.751636 20499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:39.751797 20499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:33:39.761015 20499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:39.762372 20499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:39.764180 20499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:39.777706 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:39.777801 20507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:39.777829 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:39.778829 20499 server_base.cc:1034] running on GCE node
I20260620 02:33:39.779524 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:39.780512 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:39.781736 20499 hybrid_clock.cc:648] HybridClock initialized: now 1781922819781668 us; error 59 us; skew 500 ppm
I20260620 02:33:39.782099 20499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:39.784530 20499 webserver.cc:460] Webserver started at http://127.0.97.2:34625/ using document root <none> and password file <none>
I20260620 02:33:39.785367 20499 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:39.785485 20499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:39.785838 20499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:39.788856 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a"
format_stamp: "Formatted at 2026-06-20 02:33:39 on dist-test-slave-r706"
server_key: "4161a3c86c9f8c3b97ad4383a2b39daf"
server_key_iv: "923d34a65021b4ee0f31335cd3f0753e"
server_key_version: "encryptionkey@0"
I20260620 02:33:39.789762 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a"
format_stamp: "Formatted at 2026-06-20 02:33:39 on dist-test-slave-r706"
server_key: "4161a3c86c9f8c3b97ad4383a2b39daf"
server_key_iv: "923d34a65021b4ee0f31335cd3f0753e"
server_key_version: "encryptionkey@0"
I20260620 02:33:39.795033 20499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:33:39.798638 20513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:39.800159 20499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:33:39.800347 20499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal
uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a"
format_stamp: "Formatted at 2026-06-20 02:33:39 on dist-test-slave-r706"
server_key: "4161a3c86c9f8c3b97ad4383a2b39daf"
server_key_iv: "923d34a65021b4ee0f31335cd3f0753e"
server_key_version: "encryptionkey@0"
I20260620 02:33:39.800594 20499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:39.870790 20499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:39.871673 20499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:39.871977 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:39.873121 20499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:39.875023 20499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:39.875115 20499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:39.875244 20499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:39.875355 20499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:39.910279 20499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43251
I20260620 02:33:39.910370 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43251 every 8 connection(s)
I20260620 02:33:39.911592 20499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:33:39.915625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20499
I20260620 02:33:39.915800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:33:39.916420 388 external_mini_cluster.cc:1428] Setting key 6b4b89e246b5a611bd8769a98899b785
I20260620 02:33:39.919714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37413
--builtin_ntp_servers=127.0.97.20:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:33:39.926083 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:39.914234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56421 (local address 127.0.97.62:37413)
0620 02:33:39.914708 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:39.914714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:39.915953 (+ 1239us) server_negotiation.cc:407] Connection header received
0620 02:33:39.917980 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:39.917986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:39.918076 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:39.918215 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:39.919722 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.920510 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.923117 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.923515 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.923613 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:39.925310 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:39.925315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:39.925319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:39.925338 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:39.925377 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:39.925388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:39.925461 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:39.925739 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 02:33:39.925824 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":228,"threads_started":1}
I20260620 02:33:39.927645 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:39.914843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37413 (local address 127.0.97.2:56421)
0620 02:33:39.915743 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:39.915797 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:39.917720 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:39.918511 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:39.918524 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:39.918668 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:39.919502 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:39.919546 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.920671 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.920679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:39.922954 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:39.922973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:39.923738 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:39.923748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:39.924092 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:39.924895 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:39.924900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:39.924957 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:39.925113 (+ 156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:39.925483 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:39.925496 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:39.925507 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:33:39.925701 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:33:39.926140 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
I20260620 02:33:39.929206 20626 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37413
I20260620 02:33:39.929526 20626 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:39.930353 20626 heartbeater.cc:511] Master 127.0.97.62:37413 requested a full tablet report, sending...
I20260620 02:33:39.931928 20309 ts_manager.cc:194] Registered new tserver with Master: bd61e48bf8f0495dbd9a4f9ff34e6c2a (127.0.97.2:43251)
I20260620 02:33:39.932753 20309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:56421
W20260620 02:33:40.260047 20630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:40.260341 20630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:40.260490 20630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:40.267876 20630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:40.267976 20630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:40.268136 20630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:33:40.277660 20630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:40.278899 20630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:40.280714 20630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:40.293831 20636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:40.293879 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:40.293850 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:40.294849 20630 server_base.cc:1034] running on GCE node
I20260620 02:33:40.295428 20630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:40.296355 20630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:40.297590 20630 hybrid_clock.cc:648] HybridClock initialized: now 1781922820297539 us; error 56 us; skew 500 ppm
I20260620 02:33:40.297940 20630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:33:40.300293 20630 webserver.cc:460] Webserver started at http://127.0.97.3:43819/ using document root <none> and password file <none>
I20260620 02:33:40.300925 20630 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:40.301088 20630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:40.301384 20630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:40.304143 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data/instance:
uuid: "c258f97d8bfb4f0c813f7dcf77e449ab"
format_stamp: "Formatted at 2026-06-20 02:33:40 on dist-test-slave-r706"
server_key: "b5fcad146f2b9d5f7ef2a71a29840648"
server_key_iv: "6f045c227813a01b46cfc3ecbca8c34c"
server_key_version: "encryptionkey@0"
I20260620 02:33:40.304937 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c258f97d8bfb4f0c813f7dcf77e449ab"
format_stamp: "Formatted at 2026-06-20 02:33:40 on dist-test-slave-r706"
server_key: "b5fcad146f2b9d5f7ef2a71a29840648"
server_key_iv: "6f045c227813a01b46cfc3ecbca8c34c"
server_key_version: "encryptionkey@0"
I20260620 02:33:40.310055 20630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:33:40.313658 20644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:40.314977 20630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:33:40.315153 20630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal
uuid: "c258f97d8bfb4f0c813f7dcf77e449ab"
format_stamp: "Formatted at 2026-06-20 02:33:40 on dist-test-slave-r706"
server_key: "b5fcad146f2b9d5f7ef2a71a29840648"
server_key_iv: "6f045c227813a01b46cfc3ecbca8c34c"
server_key_version: "encryptionkey@0"
I20260620 02:33:40.315379 20630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:40.345131 20630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:40.346076 20630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:40.346366 20630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:40.347417 20630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:33:40.349128 20630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:33:40.349223 20630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:40.349368 20630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:33:40.349450 20630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:40.384284 20630 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39121
I20260620 02:33:40.384375 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39121 every 8 connection(s)
I20260620 02:33:40.385725 20630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:33:40.391320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20630
I20260620 02:33:40.391511 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:33:40.392055 388 external_mini_cluster.cc:1428] Setting key 9fd6873e4501b77554d88d3003ae2c62
I20260620 02:33:40.399914 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.388412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49145 (local address 127.0.97.62:37413)
0620 02:33:40.388716 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.388721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.390060 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 02:33:40.391903 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.391909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.391999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.392132 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:40.393472 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.394410 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.396744 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.397431 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.397570 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.399059 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:40.399066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:40.399070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:40.399092 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:40.399133 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:40.399150 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:40.399247 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.399502 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.399599 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 02:33:40.400821 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.388953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37413 (local address 127.0.97.3:49145)
0620 02:33:40.389927 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.389963 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:40.391631 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:40.392357 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:40.392365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:40.392448 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:33:40.393252 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.393278 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.394575 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.394580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.396598 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.396610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.397671 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.397677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.397899 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.398711 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:33:40.398714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:33:40.398762 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:33:40.398847 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:33:40.399282 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:33:40.399291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:33:40.399298 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:33:40.399494 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 02:33:40.399786 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":284,"threads_started":1}
I20260620 02:33:40.402204 20757 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37413
I20260620 02:33:40.402451 20757 heartbeater.cc:464] Registering TS with master...
I20260620 02:33:40.403079 20757 heartbeater.cc:511] Master 127.0.97.62:37413 requested a full tablet report, sending...
I20260620 02:33:40.404703 20309 ts_manager.cc:194] Registered new tserver with Master: c258f97d8bfb4f0c813f7dcf77e449ab (127.0.97.3:39121)
I20260620 02:33:40.405784 20309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49145
I20260620 02:33:40.407137 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:33:40.417447 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.411444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.0.97.62:37413)
0620 02:33:40.411619 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.411623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.412086 (+ 463us) server_negotiation.cc:407] Connection header received
0620 02:33:40.412272 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.412276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.412333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.412418 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:40.412976 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.413999 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.415299 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.415704 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.415793 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.416753 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:40.416757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:40.416760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:40.416776 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:40.416804 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:40.416816 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:40.416884 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.416976 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.417117 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:33:40.423744 20495 heartbeater.cc:503] Master 127.0.97.62:37413 was elected leader, sending a full tablet report...
I20260620 02:33:40.421270 20309 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:33:40.428390 20309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:33:40.455066 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.441410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.0.97.3:39121)
0620 02:33:40.442191 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.442199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.442975 (+ 776us) server_negotiation.cc:407] Connection header received
0620 02:33:40.443444 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.443457 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.443778 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.443975 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:40.445355 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.447549 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.452023 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.453563 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.453695 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.453993 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.454504 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.454685 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":559,"threads_started":1}
I20260620 02:33:40.455154 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.440256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39121 (local address 127.0.0.1:47714)
0620 02:33:40.442819 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.442864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:40.443166 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:40.444453 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:40.444465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:40.444507 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:40.445087 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.445101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.447801 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.447809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.451711 (+ 3902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.451730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.454129 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.454134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.454269 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.454300 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:33:40.454434 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 02:33:40.454628 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2295,"mutex_wait_us":207,"spinlock_wait_cycles":49408,"thread_start_us":2076,"threads_started":1}
I20260620 02:33:40.456074 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.440818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.0.97.2:43251)
0620 02:33:40.441732 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.441746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.442976 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 02:33:40.444649 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.444660 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.444939 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.445155 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:40.446323 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.448373 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.453055 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.454527 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.454657 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.454890 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.455530 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.455718 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":942,"threads_started":1}
I20260620 02:33:40.456806 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.440333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43251 (local address 127.0.0.1:33704)
0620 02:33:40.442819 (+ 2486us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.442864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:40.444383 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:40.445562 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:40.445569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:40.445603 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:40.446097 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.446111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.449969 (+ 3858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.449977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.452738 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.452757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.455048 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.455055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.455304 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.455339 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:33:40.456284 (+ 945us) client_negotiation.cc:240] Negotiation successful
0620 02:33:40.456417 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2278,"mutex_wait_us":1,"spinlock_wait_cycles":14336,"thread_start_us":2071,"threads_started":1}
I20260620 02:33:40.459646 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.443271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37724)
0620 02:33:40.451844 (+ 8573us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.451860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:40.452005 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:40.452561 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:40.452566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:40.452594 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:40.453092 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.453106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.455072 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.455084 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.457277 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.457289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.458953 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.458958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.459126 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.459154 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:33:40.459240 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:33:40.459363 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8042,"thread_start_us":7775,"threads_started":1}
I20260620 02:33:40.459244 20692 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a3d2f170c40eca85b252f2f1a257f (DEFAULT_TABLE table=test-table [id=88423feeae5f460aa945dda67766891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:40.460286 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.443515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.0.97.1:42663)
0620 02:33:40.451666 (+ 8151us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.451675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.451959 (+ 284us) server_negotiation.cc:407] Connection header received
0620 02:33:40.452163 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.452186 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.452372 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.452570 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:40.453314 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.454909 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.457455 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.458749 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.458893 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.459151 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.459652 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.459930 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7971,"thread_start_us":7818,"threads_started":1}
I20260620 02:33:40.460454 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a3d2f170c40eca85b252f2f1a257f (DEFAULT_TABLE table=test-table [id=88423feeae5f460aa945dda67766891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:40.461221 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a3d2f170c40eca85b252f2f1a257f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:40.461732 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a3d2f170c40eca85b252f2f1a257f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:40.463754 20430 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a3d2f170c40eca85b252f2f1a257f (DEFAULT_TABLE table=test-table [id=88423feeae5f460aa945dda67766891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:33:40.465034 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a3d2f170c40eca85b252f2f1a257f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:40.473790 20775 tablet_bootstrap.cc:492] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Bootstrap starting.
I20260620 02:33:40.477166 20775 tablet_bootstrap.cc:654] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:40.477371 20776 tablet_bootstrap.cc:492] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Bootstrap starting.
I20260620 02:33:40.478612 20775 log.cc:824] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:40.479746 20777 tablet_bootstrap.cc:492] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a: Bootstrap starting.
I20260620 02:33:40.481783 20776 tablet_bootstrap.cc:654] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:40.483251 20777 tablet_bootstrap.cc:654] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:40.483213 20776 log.cc:824] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:40.484411 20777 log.cc:824] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:40.484477 20775 tablet_bootstrap.cc:492] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: No bootstrap required, opened a new log
I20260620 02:33:40.484822 20775 ts_tablet_manager.cc:1397] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260620 02:33:40.486721 20776 tablet_bootstrap.cc:492] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: No bootstrap required, opened a new log
I20260620 02:33:40.486955 20777 tablet_bootstrap.cc:492] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a: No bootstrap required, opened a new log
I20260620 02:33:40.487001 20776 ts_tablet_manager.cc:1397] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:33:40.487203 20777 ts_tablet_manager.cc:1397] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 02:33:40.496476 20775 raft_consensus.cc:348] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.497042 20775 raft_consensus.cc:374] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:40.497188 20775 raft_consensus.cc:729] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd61e48bf8f0495dbd9a4f9ff34e6c2a, State: Initialized, Role: FOLLOWER
I20260620 02:33:40.497845 20776 raft_consensus.cc:348] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.498052 20775 consensus_queue.cc:260] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.498484 20776 raft_consensus.cc:374] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:40.498647 20776 raft_consensus.cc:729] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c258f97d8bfb4f0c813f7dcf77e449ab, State: Initialized, Role: FOLLOWER
I20260620 02:33:40.499456 20626 heartbeater.cc:503] Master 127.0.97.62:37413 was elected leader, sending a full tablet report...
I20260620 02:33:40.500108 20776 consensus_queue.cc:260] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.500013 20777 raft_consensus.cc:348] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.500587 20777 raft_consensus.cc:374] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:40.500738 20777 raft_consensus.cc:729] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497c2ef0904d42eba65d161d46269a1a, State: Initialized, Role: FOLLOWER
I20260620 02:33:40.501463 20777 consensus_queue.cc:260] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.502123 20757 heartbeater.cc:503] Master 127.0.97.62:37413 was elected leader, sending a full tablet report...
I20260620 02:33:40.502844 20776 ts_tablet_manager.cc:1428] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260620 02:33:40.504177 20775 ts_tablet_manager.cc:1428] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260620 02:33:40.505928 20777 ts_tablet_manager.cc:1428] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a: Time spent starting tablet: real 0.019s user 0.014s sys 0.001s
I20260620 02:33:40.563797 20782 raft_consensus.cc:482] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:33:40.564152 20782 raft_consensus.cc:504] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.565970 20782 leader_election.cc:288] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd61e48bf8f0495dbd9a4f9ff34e6c2a (127.0.97.2:43251), 497c2ef0904d42eba65d161d46269a1a (127.0.97.1:42663)
I20260620 02:33:40.572683 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.566460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45037 (local address 127.0.97.2:43251)
0620 02:33:40.566654 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.566660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.566684 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:33:40.567011 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.567018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.567122 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.567230 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:40.567837 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.568830 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.570759 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.571716 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.571814 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.571881 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.572258 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.572375 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:33:40.572723 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.566380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43251 (local address 127.0.97.3:45037)
0620 02:33:40.566556 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.566607 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:40.566775 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:40.567331 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:40.567337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:40.567367 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:40.567603 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.567616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.568988 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.569021 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.570600 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.570613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.571873 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.571880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.572065 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.572106 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:33:40.572239 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 02:33:40.572395 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":2}
I20260620 02:33:40.573272 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.566505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60133 (local address 127.0.97.1:42663)
0620 02:33:40.566733 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.566741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.567464 (+ 723us) server_negotiation.cc:407] Connection header received
0620 02:33:40.567664 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.567672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.567772 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.567867 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:33:40.568494 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.569372 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.571256 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.572293 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.572403 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.572510 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.572882 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.573036 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:33:40.573956 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.566380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.97.3:60133)
0620 02:33:40.567345 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.567365 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:33:40.567503 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:33:40.567886 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:33:40.567894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:33:40.567922 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:33:40.568261 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.568272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.569516 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.569520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.571078 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:33:40.571093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.572475 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.572482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:33:40.572664 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.572701 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:33:40.573448 (+ 747us) client_negotiation.cc:240] Negotiation successful
0620 02:33:40.573589 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":209,"thread_start_us":372,"threads_started":1}
I20260620 02:33:40.574195 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a3d2f170c40eca85b252f2f1a257f" candidate_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" is_pre_election: true
I20260620 02:33:40.574741 20581 raft_consensus.cc:2454] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c258f97d8bfb4f0c813f7dcf77e449ab in term 0.
I20260620 02:33:40.574970 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a3d2f170c40eca85b252f2f1a257f" candidate_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "497c2ef0904d42eba65d161d46269a1a" is_pre_election: true
I20260620 02:33:40.575456 20648 leader_election.cc:302] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd61e48bf8f0495dbd9a4f9ff34e6c2a, c258f97d8bfb4f0c813f7dcf77e449ab; no voters:
I20260620 02:33:40.575555 20450 raft_consensus.cc:2454] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c258f97d8bfb4f0c813f7dcf77e449ab in term 0.
I20260620 02:33:40.575882 20782 raft_consensus.cc:2781] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:33:40.576037 20782 raft_consensus.cc:482] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:33:40.576166 20782 raft_consensus.cc:3037] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:40.578490 20782 raft_consensus.cc:504] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.579088 20782 leader_election.cc:288] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [CANDIDATE]: Term 1 election: Requested vote from peers bd61e48bf8f0495dbd9a4f9ff34e6c2a (127.0.97.2:43251), 497c2ef0904d42eba65d161d46269a1a (127.0.97.1:42663)
I20260620 02:33:40.579586 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a3d2f170c40eca85b252f2f1a257f" candidate_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "497c2ef0904d42eba65d161d46269a1a"
I20260620 02:33:40.579597 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a3d2f170c40eca85b252f2f1a257f" candidate_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a"
I20260620 02:33:40.579790 20450 raft_consensus.cc:3037] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:40.579810 20581 raft_consensus.cc:3037] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:40.582120 20581 raft_consensus.cc:2454] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c258f97d8bfb4f0c813f7dcf77e449ab in term 1.
I20260620 02:33:40.582118 20450 raft_consensus.cc:2454] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c258f97d8bfb4f0c813f7dcf77e449ab in term 1.
I20260620 02:33:40.582556 20648 leader_election.cc:302] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd61e48bf8f0495dbd9a4f9ff34e6c2a, c258f97d8bfb4f0c813f7dcf77e449ab; no voters:
I20260620 02:33:40.582873 20782 raft_consensus.cc:2781] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:40.583357 20782 raft_consensus.cc:686] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 1 LEADER]: Becoming Leader. State: Replica: c258f97d8bfb4f0c813f7dcf77e449ab, State: Running, Role: LEADER
I20260620 02:33:40.583811 20782 consensus_queue.cc:237] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } }
I20260620 02:33:40.588380 20308 catalog_manager.cc:5495] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c258f97d8bfb4f0c813f7dcf77e449ab (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c258f97d8bfb4f0c813f7dcf77e449ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 } health_report { overall_health: UNKNOWN } } }
I20260620 02:33:40.610759 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.605356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.0.97.3:39121)
0620 02:33:40.605584 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.605591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.605667 (+ 76us) server_negotiation.cc:407] Connection header received
0620 02:33:40.605848 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.605852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.605930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.606069 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:33:40.606375 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.607105 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.608469 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.608941 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.609097 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.609546 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:33:40.610170 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:33:40.610282 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.610410 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.610475 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:33:40.618799 20581 raft_consensus.cc:1261] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 1 FOLLOWER]: Refusing update from remote peer c258f97d8bfb4f0c813f7dcf77e449ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:40.618772 20450 raft_consensus.cc:1261] T 8d8a3d2f170c40eca85b252f2f1a257f P 497c2ef0904d42eba65d161d46269a1a [term 1 FOLLOWER]: Refusing update from remote peer c258f97d8bfb4f0c813f7dcf77e449ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:33:40.619807 20785 consensus_queue.cc:1035] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "497c2ef0904d42eba65d161d46269a1a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:40.620143 20782 consensus_queue.cc:1035] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd61e48bf8f0495dbd9a4f9ff34e6c2a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:33:40.646682 20307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:40.646991 20307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:33:40.650514 20307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782b948feb55493b88b240afe2d71853: Sending DeleteTablet for 3 replicas of tablet 8d8a3d2f170c40eca85b252f2f1a257f
I20260620 02:33:40.651604 20561 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8a3d2f170c40eca85b252f2f1a257f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:40 UTC) from {username='slave'} at 127.0.0.1:33704
I20260620 02:33:40.651616 20692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8a3d2f170c40eca85b252f2f1a257f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:40 UTC) from {username='slave'} at 127.0.0.1:47714
I20260620 02:33:40.652603 20799 tablet_replica.cc:330] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: stopping tablet replica
I20260620 02:33:40.652933 20430 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8a3d2f170c40eca85b252f2f1a257f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:33:40 UTC) from {username='slave'} at 127.0.0.1:37724
I20260620 02:33:40.653205 20798 tablet_replica.cc:330] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: stopping tablet replica
I20260620 02:33:40.653613 20798 raft_consensus.cc:2229] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:33:40.654146 20798 raft_consensus.cc:2258] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:40.654402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20368
I20260620 02:33:40.654817 20799 raft_consensus.cc:2229] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:33:40.655628 20799 raft_consensus.cc:2258] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:33:40.658361 20799 ts_tablet_manager.cc:1905] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:40.659691 20798 ts_tablet_manager.cc:1905] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:33:40.665385 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.656900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.0.97.1:42663)
0620 02:33:40.657129 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.657136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.657155 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:33:40.657219 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.657224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.657343 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.657450 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:40.658102 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.659311 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.661210 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.661890 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.661989 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.663588 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:40.663602 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:40.663621 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:40.664402 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:40.664462 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:40.664488 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:40.664580 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.664847 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.664963 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:33:40.666581 20490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:40.668468 20798 ts_tablet_manager.cc:1918] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:40.668665 20798 log.cc:1196] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-1/wal/wals/8d8a3d2f170c40eca85b252f2f1a257f
I20260620 02:33:40.669296 20798 ts_tablet_manager.cc:1939] T 8d8a3d2f170c40eca85b252f2f1a257f P bd61e48bf8f0495dbd9a4f9ff34e6c2a: Deleting consensus metadata
I20260620 02:33:40.671356 20296 catalog_manager.cc:4841] TS bd61e48bf8f0495dbd9a4f9ff34e6c2a (127.0.97.2:43251): tablet 8d8a3d2f170c40eca85b252f2f1a257f (table test-table [id=88423feeae5f460aa945dda67766891e]) successfully deleted
I20260620 02:33:40.671741 20799 ts_tablet_manager.cc:1918] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:33:40.671957 20799 log.cc:1196] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Deleting WAL directory at /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922485472743-388-0/minicluster-data/ts-2/wal/wals/8d8a3d2f170c40eca85b252f2f1a257f
I20260620 02:33:40.672503 20799 ts_tablet_manager.cc:1939] T 8d8a3d2f170c40eca85b252f2f1a257f P c258f97d8bfb4f0c813f7dcf77e449ab: Deleting consensus metadata
I20260620 02:33:40.674494 20294 catalog_manager.cc:4841] TS c258f97d8bfb4f0c813f7dcf77e449ab (127.0.97.3:39121): tablet 8d8a3d2f170c40eca85b252f2f1a257f (table test-table [id=88423feeae5f460aa945dda67766891e]) successfully deleted
I20260620 02:33:40.783905 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20368
I20260620 02:33:40.802507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20499
W20260620 02:33:40.802528 20294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42663 (error 108)
I20260620 02:33:40.804539 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.804085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37734)
0620 02:33:40.804291 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.804405 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:33:40.804930 20294 catalog_manager.cc:4584] TS 497c2ef0904d42eba65d161d46269a1a (127.0.97.1:42663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d8a3d2f170c40eca85b252f2f1a257f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
I20260620 02:33:40.811131 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.803626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.0.97.2:43251)
0620 02:33:40.803803 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.803809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.803833 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:33:40.803904 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.803907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.804068 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.804184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:40.804787 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.805973 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.807468 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.808125 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.808261 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.809483 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:40.809499 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:40.809522 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:40.810301 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:40.810365 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:40.810393 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:40.810521 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.810664 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.810777 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:33:40.811813 20621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:40.858733 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.858146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37746)
0620 02:33:40.858401 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.858520 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 02:33:40.859980 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.859555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37750)
0620 02:33:40.859721 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.859816 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 02:33:40.903673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20499
I20260620 02:33:40.920948 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.920378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37756)
0620 02:33:40.920615 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.920744 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:33:40.922037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20630
I20260620 02:33:40.922561 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.922098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37768)
0620 02:33:40.922310 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:40.922375 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 02:33:40.930284 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:40.922786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.0.97.3:39121)
0620 02:33:40.922968 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:40.922976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:40.923092 (+ 116us) server_negotiation.cc:407] Connection header received
0620 02:33:40.923225 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:40.923231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:40.923326 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:40.923446 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:40.923867 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.925096 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.926492 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:40.927150 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:40.927292 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:40.928392 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:40.928402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:40.928418 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:40.929400 (+ 982us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:40.929479 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:40.929510 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:40.929635 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:40.929773 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 02:33:40.929888 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:33:40.931348 20752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:41.003587 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:41.002992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37784)
0620 02:33:41.003264 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:41.003380 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 02:33:41.004786 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:41.004327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37792)
0620 02:33:41.004548 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:41.004653 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 02:33:41.055543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20630
I20260620 02:33:41.076651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20278
I20260620 02:33:41.077813 20339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:41.188884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20278
I20260620 02:33:41.188942 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:41.187404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42663 (local address 127.0.0.1:37798)
0620 02:33:41.188182 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 02:33:41.188302 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590}
2026-06-20T02:33:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2891 ms)
[----------] 6 tests from EncryptionPolicyTest (22751 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:41 dist-test-slave-r706 krb5kdc[20809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:41 dist-test-slave-r706 krb5kdc[20809](info): set up 2 sockets
Jun 20 02:33:41 dist-test-slave-r706 krb5kdc[20809](info): commencing operation
krb5kdc: starting...
W20260620 02:33:42.321591 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:42 dist-test-slave-r706 krb5kdc[20809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:42.620427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:42.979945 20822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:42.980257 20822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:42.980407 20822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:42.988086 20822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:42.988214 20822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:42.988302 20822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:42.988417 20822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:42.988504 20822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:42.998840 20822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:43.000205 20822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:43.001574 20822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:43.013360 20827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:43.013463 20830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:43.013417 20828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:43.014420 20822 server_base.cc:1034] running on GCE node
I20260620 02:33:43.015267 20822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:43.016770 20822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:43.018042 20822 hybrid_clock.cc:648] HybridClock initialized: now 1781922823017953 us; error 80 us; skew 500 ppm
I20260620 02:33:43.018424 20822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:43 dist-test-slave-r706 krb5kdc[20809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:43.021561 20822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:43.023103 20822 webserver.cc:460] Webserver started at http://127.0.0.1:36691/ using document root <none> and password file <none>
I20260620 02:33:43.023823 20822 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:43.023993 20822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:43.024324 20822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:43.027616 20822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "fe9b689023b242bd99abf661fb21d08d"
format_stamp: "Formatted at 2026-06-20 02:33:43 on dist-test-slave-r706"
server_key: "1acca3422b5ec8105d198614e0de5fb9"
server_key_iv: "983950633095ec21dec4d4613841abf8"
server_key_version: "encryptionkey@0"
I20260620 02:33:43.028606 20822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "fe9b689023b242bd99abf661fb21d08d"
format_stamp: "Formatted at 2026-06-20 02:33:43 on dist-test-slave-r706"
server_key: "1acca3422b5ec8105d198614e0de5fb9"
server_key_iv: "983950633095ec21dec4d4613841abf8"
server_key_version: "encryptionkey@0"
I20260620 02:33:43.034121 20822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:33:43.037714 20837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:43.039072 20822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:43.039263 20822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "fe9b689023b242bd99abf661fb21d08d"
format_stamp: "Formatted at 2026-06-20 02:33:43 on dist-test-slave-r706"
server_key: "1acca3422b5ec8105d198614e0de5fb9"
server_key_iv: "983950633095ec21dec4d4613841abf8"
server_key_version: "encryptionkey@0"
I20260620 02:33:43.039465 20822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:43.054674 20822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:43.059077 20822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:43.059388 20822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:43.078964 20822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39427
I20260620 02:33:43.079056 20888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39427 every 8 connection(s)
I20260620 02:33:43.080475 20822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:43.084352 20889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:43.089471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20822
I20260620 02:33:43.089705 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:43.090432 388 external_mini_cluster.cc:1428] Setting key 30e689680174e23a7733ac3ecaf47593
I20260620 02:33:43.096773 20889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d: Bootstrap starting.
I20260620 02:33:43.100829 20889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:43.102270 20889 log.cc:824] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:43.106215 20889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d: No bootstrap required, opened a new log
Jun 20 02:33:43 dist-test-slave-r706 krb5kdc[20809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:33:43.114820 20889 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b689023b242bd99abf661fb21d08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39427 } }
I20260620 02:33:43.115298 20889 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:43.115454 20889 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9b689023b242bd99abf661fb21d08d, State: Initialized, Role: FOLLOWER
I20260620 02:33:43.116077 20889 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b689023b242bd99abf661fb21d08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39427 } }
I20260620 02:33:43.116309 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:43.092661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47529 (local address 127.0.0.1:39427)
0620 02:33:43.093640 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:43.093655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:43.093700 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:33:43.095508 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:43.095604 (+ 96us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:43.096034 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:43.096423 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:43.098377 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:43.099779 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:43.101838 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:43.102682 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:43.102905 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:43.108004 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:43.108012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:43.108029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:43.108094 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:43.111476 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:43.112442 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:43.112448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:43.112456 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:43.112635 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:43.113135 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:43.113139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:43.113141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:43.113764 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:43.114133 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:43.114622 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 02:33:43.114954 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":509,"threads_started":1}
I20260620 02:33:43.116413 20889 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:43.116569 20889 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:43.116763 20889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:43.119443 20889 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b689023b242bd99abf661fb21d08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39427 } }
I20260620 02:33:43.120139 20889 leader_election.cc:302] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe9b689023b242bd99abf661fb21d08d; no voters:
I20260620 02:33:43.120568 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:43.120833 20889 leader_election.cc:288] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:43.120959 20894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:43.122849 20889 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:43.130092 20894 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [term 1 LEADER]: Becoming Leader. State: Replica: fe9b689023b242bd99abf661fb21d08d, State: Running, Role: LEADER
I20260620 02:33:43.131109 20894 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b689023b242bd99abf661fb21d08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39427 } }
I20260620 02:33:43.135519 20895 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe9b689023b242bd99abf661fb21d08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b689023b242bd99abf661fb21d08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39427 } } }
I20260620 02:33:43.135959 20895 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:43.136490 20896 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe9b689023b242bd99abf661fb21d08d. Latest consensus state: current_term: 1 leader_uuid: "fe9b689023b242bd99abf661fb21d08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b689023b242bd99abf661fb21d08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39427 } } }
I20260620 02:33:43.136787 20896 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:43.138799 20913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:43.144254 20913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:43.150864 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:43.132826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36695 (local address 127.0.0.1:39427)
0620 02:33:43.133055 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:43.133064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:43.133700 (+ 636us) server_negotiation.cc:407] Connection header received
0620 02:33:43.134022 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:43.134030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:43.134141 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:43.134221 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:43.138038 (+ 3817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:43.139290 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:43.140885 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:43.141429 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:43.141598 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:43.144467 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:43.144472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:43.144475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:43.144555 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:43.147701 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:43.148544 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:43.148550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:43.148554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:43.148694 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:43.149238 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:43.149242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:43.149244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:43.149592 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:43.149866 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:43.150265 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 02:33:43.150471 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:33:43.154574 20913 catalog_manager.cc:1347] Generated new cluster ID: 178566e9f87f4f589aa734da948a148b
I20260620 02:33:43.154697 20913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:43.168834 20913 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:43.170195 20913 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:43.181113 20913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe9b689023b242bd99abf661fb21d08d: Generated new TSK 0
I20260620 02:33:43.182185 20913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:43.196520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20822
I20260620 02:33:43.198067 20884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:43.290140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20822
W20260620 02:33:43.310107 20807 connection.cc:535] client connection to 127.0.0.1:39427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39427): Connection reset by peer (error 104)
2026-06-20T02:33:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2093 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:43 dist-test-slave-r706 krb5kdc[20921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:43 dist-test-slave-r706 krb5kdc[20921](info): set up 2 sockets
Jun 20 02:33:43 dist-test-slave-r706 krb5kdc[20921](info): commencing operation
krb5kdc: starting...
W20260620 02:33:44.413486 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:44 dist-test-slave-r706 krb5kdc[20921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:44.718899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:45.083221 20934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:45.083561 20934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:45.083658 20934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:45.091266 20934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:45.091411 20934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:45.091460 20934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:45.091522 20934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:45.091569 20934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:45.102069 20934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:45.103431 20934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:45.104801 20934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:45.118386 20939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:45.118438 20942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:45.118460 20940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:45.119616 20934 server_base.cc:1034] running on GCE node
I20260620 02:33:45.120561 20934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:45.122262 20934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:45.123507 20934 hybrid_clock.cc:648] HybridClock initialized: now 1781922825123465 us; error 49 us; skew 500 ppm
I20260620 02:33:45.123888 20934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:45 dist-test-slave-r706 krb5kdc[20921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:45.133066 20934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:45.134902 20934 webserver.cc:460] Webserver started at http://127.0.0.1:42643/ using document root <none> and password file <none>
I20260620 02:33:45.135685 20934 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:45.135844 20934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:45.136188 20934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:45.139489 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "52b80c4df3444b78940fedfe87cb512f"
format_stamp: "Formatted at 2026-06-20 02:33:45 on dist-test-slave-r706"
server_key: "c97892748566da84c9c3f74da28f5ce0"
server_key_iv: "797826ced23f6c573e84bd6991466cfd"
server_key_version: "encryptionkey@0"
I20260620 02:33:45.140367 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "52b80c4df3444b78940fedfe87cb512f"
format_stamp: "Formatted at 2026-06-20 02:33:45 on dist-test-slave-r706"
server_key: "c97892748566da84c9c3f74da28f5ce0"
server_key_iv: "797826ced23f6c573e84bd6991466cfd"
server_key_version: "encryptionkey@0"
I20260620 02:33:45.146633 20934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:33:45.151031 20949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:45.152632 20934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 02:33:45.152864 20934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "52b80c4df3444b78940fedfe87cb512f"
format_stamp: "Formatted at 2026-06-20 02:33:45 on dist-test-slave-r706"
server_key: "c97892748566da84c9c3f74da28f5ce0"
server_key_iv: "797826ced23f6c573e84bd6991466cfd"
server_key_version: "encryptionkey@0"
I20260620 02:33:45.153198 20934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:45.170341 20934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:45.175275 20934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:45.175628 20934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:45.196364 20934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40605
I20260620 02:33:45.196426 21000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40605 every 8 connection(s)
I20260620 02:33:45.197861 20934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:45.200098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 20934
I20260620 02:33:45.200330 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:45.200912 388 external_mini_cluster.cc:1428] Setting key e352b85eaf4cf0aee3e9dd6788a576ca
I20260620 02:33:45.202749 21001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:45.215860 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:45.202859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59359 (local address 127.0.0.1:40605)
0620 02:33:45.203745 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:45.203757 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:45.203800 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:33:45.205532 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:45.205554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:45.205959 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:45.206308 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:45.206937 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:45.208314 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:45.210322 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:45.211388 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:45.211549 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:45.213248 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:45.213257 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:45.213274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:45.213893 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:45.213974 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:45.214005 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:45.214160 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:45.214581 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 02:33:45.214798 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":383,"threads_started":1}
I20260620 02:33:45.216663 21001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f: Bootstrap starting.
I20260620 02:33:45.220098 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:45.220196 21001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:45.221586 21001 log.cc:824] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:45.225467 21001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f: No bootstrap required, opened a new log
I20260620 02:33:45.233266 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:45.225831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46797 (local address 127.0.0.1:40605)
0620 02:33:45.226075 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:45.226081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:45.226486 (+ 405us) server_negotiation.cc:407] Connection header received
0620 02:33:45.226797 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:45.226804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:45.226899 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:45.227025 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:45.227541 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:45.228768 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:45.230439 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:45.230926 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:45.231113 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:45.232355 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:45.232361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:45.232364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:45.232384 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:45.232428 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:45.232444 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:45.232590 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:45.232803 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 02:33:45.232865 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:33:45.234704 21001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b80c4df3444b78940fedfe87cb512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260620 02:33:45.235262 21001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:45.235383 21001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b80c4df3444b78940fedfe87cb512f, State: Initialized, Role: FOLLOWER
I20260620 02:33:45.236011 21001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b80c4df3444b78940fedfe87cb512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260620 02:33:45.236279 21001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:45.236439 21001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:45.236655 21001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:45.239246 21001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b80c4df3444b78940fedfe87cb512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260620 02:33:45.239817 21001 leader_election.cc:302] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52b80c4df3444b78940fedfe87cb512f; no voters:
I20260620 02:33:45.240450 21001 leader_election.cc:288] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:45.240635 21011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:45.241415 21011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [term 1 LEADER]: Becoming Leader. State: Replica: 52b80c4df3444b78940fedfe87cb512f, State: Running, Role: LEADER
I20260620 02:33:45.242031 21011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b80c4df3444b78940fedfe87cb512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260620 02:33:45.242820 21001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:45.245585 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52b80c4df3444b78940fedfe87cb512f. Latest consensus state: current_term: 1 leader_uuid: "52b80c4df3444b78940fedfe87cb512f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b80c4df3444b78940fedfe87cb512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } } }
I20260620 02:33:45.245564 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52b80c4df3444b78940fedfe87cb512f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b80c4df3444b78940fedfe87cb512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } } }
I20260620 02:33:45.246121 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:45.246187 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:45.247079 21016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:45.253829 21016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:45.264117 21016 catalog_manager.cc:1347] Generated new cluster ID: 5d140da338a14c9f9a6b9b911a129333
I20260620 02:33:45.264266 21016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:45.283002 21016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:45.284870 21016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:45.306576 21016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52b80c4df3444b78940fedfe87cb512f: Generated new TSK 0
I20260620 02:33:45.307818 21016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:45.313896 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20934
W20260620 02:33:45.315079 20996 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59359
W20260620 02:33:45.315872 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 20934: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:33:45.316071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 20934
W20260620 02:33:45.332350 20919 connection.cc:535] client connection to 127.0.0.1:40605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40605): Connection reset by peer (error 104)
2026-06-20T02:33:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:45 dist-test-slave-r706 krb5kdc[21032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:45 dist-test-slave-r706 krb5kdc[21032](info): set up 2 sockets
Jun 20 02:33:45 dist-test-slave-r706 krb5kdc[21032](info): commencing operation
krb5kdc: starting...
W20260620 02:33:46.478163 388 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:46 dist-test-slave-r706 krb5kdc[21032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:46.784823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:47.143983 21045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:47.144296 21045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:47.144474 21045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:47.153110 21045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:47.153318 21045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:47.153411 21045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:47.153507 21045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:47.153596 21045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:47.164126 21045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:47.165614 21045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:47.166965 21045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:47.179515 21050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:47.179549 21051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:47.179589 21053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:47.180547 21045 server_base.cc:1034] running on GCE node
I20260620 02:33:47.181558 21045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:47.183048 21045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:47.184270 21045 hybrid_clock.cc:648] HybridClock initialized: now 1781922827184221 us; error 39 us; skew 500 ppm
I20260620 02:33:47.184687 21045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:47 dist-test-slave-r706 krb5kdc[21032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:47.188218 21045 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:47.190004 21045 webserver.cc:460] Webserver started at http://127.0.0.1:41157/ using document root <none> and password file <none>
I20260620 02:33:47.190749 21045 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:47.190909 21045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:47.191246 21045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:47.194657 21045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "e548b211e0db405f95b00695b054e92d"
format_stamp: "Formatted at 2026-06-20 02:33:47 on dist-test-slave-r706"
server_key: "0e9a2e0e231e60bba47a6f00773b7d33"
server_key_iv: "1838bcfd7027e6d5b0f9bd5e1b1389b7"
server_key_version: "encryptionkey@0"
I20260620 02:33:47.195533 21045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "e548b211e0db405f95b00695b054e92d"
format_stamp: "Formatted at 2026-06-20 02:33:47 on dist-test-slave-r706"
server_key: "0e9a2e0e231e60bba47a6f00773b7d33"
server_key_iv: "1838bcfd7027e6d5b0f9bd5e1b1389b7"
server_key_version: "encryptionkey@0"
I20260620 02:33:47.201282 21045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:33:47.205106 21060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:47.206516 21045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:47.206745 21045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "e548b211e0db405f95b00695b054e92d"
format_stamp: "Formatted at 2026-06-20 02:33:47 on dist-test-slave-r706"
server_key: "0e9a2e0e231e60bba47a6f00773b7d33"
server_key_iv: "1838bcfd7027e6d5b0f9bd5e1b1389b7"
server_key_version: "encryptionkey@0"
I20260620 02:33:47.206975 21045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:47.213301 21045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:47.217669 21045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:47.217998 21045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:47.237782 21045 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44319
I20260620 02:33:47.237830 21111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44319 every 8 connection(s)
I20260620 02:33:47.239295 21045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:47.243126 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:47.245289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21045
I20260620 02:33:47.245527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:47.246078 388 external_mini_cluster.cc:1428] Setting key 24b0042409344a918e50452a5d115719
I20260620 02:33:47.254757 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:47.248054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41169 (local address 127.0.0.1:44319)
0620 02:33:47.249101 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:47.249114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:47.249163 (+ 49us) server_negotiation.cc:407] Connection header received
0620 02:33:47.251023 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:47.251047 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:47.251401 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:47.251813 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:47.252961 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:47.252968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:47.252986 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:47.253047 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:47.253123 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:47.253154 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:47.253360 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:47.253477 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 02:33:47.253737 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":474,"threads_started":1}
I20260620 02:33:47.256438 21112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d: Bootstrap starting.
I20260620 02:33:47.259109 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:47.259930 21112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:47.260972 21112 log.cc:824] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:47.264586 21112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d: No bootstrap required, opened a new log
I20260620 02:33:47.266289 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:47.263684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43525 (local address 127.0.0.1:44319)
0620 02:33:47.263877 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:47.263885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:47.264234 (+ 349us) server_negotiation.cc:407] Connection header received
0620 02:33:47.264447 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:47.264451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:47.264499 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:47.264619 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:47.265603 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:47.265608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:47.265613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:47.265634 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:47.265677 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:47.265692 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:47.265822 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:47.265940 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 02:33:47.266004 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:33:47.272696 21112 raft_consensus.cc:348] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e548b211e0db405f95b00695b054e92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260620 02:33:47.273370 21112 raft_consensus.cc:374] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:47.273509 21112 raft_consensus.cc:729] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e548b211e0db405f95b00695b054e92d, State: Initialized, Role: FOLLOWER
I20260620 02:33:47.274142 21112 consensus_queue.cc:260] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e548b211e0db405f95b00695b054e92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260620 02:33:47.274448 21112 raft_consensus.cc:388] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:47.274608 21112 raft_consensus.cc:482] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:47.274775 21112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:47.277115 21112 raft_consensus.cc:504] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e548b211e0db405f95b00695b054e92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260620 02:33:47.277622 21112 leader_election.cc:302] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e548b211e0db405f95b00695b054e92d; no voters:
I20260620 02:33:47.278204 21112 leader_election.cc:288] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:47.278381 21122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:47.279186 21122 raft_consensus.cc:686] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [term 1 LEADER]: Becoming Leader. State: Replica: e548b211e0db405f95b00695b054e92d, State: Running, Role: LEADER
I20260620 02:33:47.279701 21122 consensus_queue.cc:237] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e548b211e0db405f95b00695b054e92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260620 02:33:47.280611 21112 sys_catalog.cc:564] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:47.283030 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e548b211e0db405f95b00695b054e92d. Latest consensus state: current_term: 1 leader_uuid: "e548b211e0db405f95b00695b054e92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e548b211e0db405f95b00695b054e92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } } }
I20260620 02:33:47.283016 21123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e548b211e0db405f95b00695b054e92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e548b211e0db405f95b00695b054e92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } } }
I20260620 02:33:47.283546 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:47.283650 21123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:47.285142 21129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:47.290542 21129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:47.300369 21129 catalog_manager.cc:1347] Generated new cluster ID: 2fe2373e2f4a48f9b149e6fe716a492b
I20260620 02:33:47.300561 21129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:47.316838 21129 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:47.317534 21129 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:47.331754 21129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e548b211e0db405f95b00695b054e92d: Generated new TSK 0
I20260620 02:33:47.332989 21129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:47.338781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21045
W20260620 02:33:47.339932 21107 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41169
W20260620 02:33:47.340684 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21045: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:33:47.340867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21045
2026-06-20T02:33:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:47 dist-test-slave-r706 krb5kdc[21143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:47 dist-test-slave-r706 krb5kdc[21143](info): set up 2 sockets
Jun 20 02:33:47 dist-test-slave-r706 krb5kdc[21143](info): commencing operation
krb5kdc: starting...
W20260620 02:33:48.501711 388 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:48 dist-test-slave-r706 krb5kdc[21143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:48.817158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:49.183408 21156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:49.183802 21156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:49.183974 21156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:49.191762 21156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:49.191964 21156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:49.192075 21156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:49.192193 21156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:49.192278 21156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:49.203877 21156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:49.205320 21156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:49.206674 21156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:49.219861 21162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:49.219807 21164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:49.219853 21161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:49.220925 21156 server_base.cc:1034] running on GCE node
I20260620 02:33:49.221849 21156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:49.223407 21156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:49.224671 21156 hybrid_clock.cc:648] HybridClock initialized: now 1781922829224611 us; error 66 us; skew 500 ppm
I20260620 02:33:49.225134 21156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:49 dist-test-slave-r706 krb5kdc[21143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:49.228407 21156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:49.229971 21156 webserver.cc:460] Webserver started at http://127.0.0.1:36903/ using document root <none> and password file <none>
I20260620 02:33:49.231329 21156 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:49.231499 21156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:49.232015 21156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:49.235092 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "01eb555031614a0caf4a3f8ab34b0cbb"
format_stamp: "Formatted at 2026-06-20 02:33:49 on dist-test-slave-r706"
server_key: "d7e180447351e11e4fdda996dd3bf875"
server_key_iv: "03db28b0a8fcddee36964fdf4b97eb92"
server_key_version: "encryptionkey@0"
I20260620 02:33:49.235849 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "01eb555031614a0caf4a3f8ab34b0cbb"
format_stamp: "Formatted at 2026-06-20 02:33:49 on dist-test-slave-r706"
server_key: "d7e180447351e11e4fdda996dd3bf875"
server_key_iv: "03db28b0a8fcddee36964fdf4b97eb92"
server_key_version: "encryptionkey@0"
I20260620 02:33:49.241811 21156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:33:49.245630 21171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:49.247069 21156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:33:49.247295 21156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "01eb555031614a0caf4a3f8ab34b0cbb"
format_stamp: "Formatted at 2026-06-20 02:33:49 on dist-test-slave-r706"
server_key: "d7e180447351e11e4fdda996dd3bf875"
server_key_iv: "03db28b0a8fcddee36964fdf4b97eb92"
server_key_version: "encryptionkey@0"
I20260620 02:33:49.247511 21156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:49.275262 21156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:49.280535 21156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:49.280892 21156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:49.301137 21156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36009
I20260620 02:33:49.301223 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36009 every 8 connection(s)
I20260620 02:33:49.302613 21156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:49.306706 21223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:49.312718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21156
I20260620 02:33:49.312944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:49.313437 388 external_mini_cluster.cc:1428] Setting key fdcbaa6e597bcb3465f783bcf711d25f
I20260620 02:33:49.318843 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb: Bootstrap starting.
I20260620 02:33:49.322263 21223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:49.323390 21223 log.cc:824] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:49.327162 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb: No bootstrap required, opened a new log
Jun 20 02:33:49 dist-test-slave-r706 krb5kdc[21143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:33:49.338511 21223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36009 } }
I20260620 02:33:49.339046 21223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:49.339161 21223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01eb555031614a0caf4a3f8ab34b0cbb, State: Initialized, Role: FOLLOWER
I20260620 02:33:49.339891 21223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36009 } }
I20260620 02:33:49.340199 21223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:49.340401 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:49.315678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51067 (local address 127.0.0.1:36009)
0620 02:33:49.316629 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:49.316641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:49.316688 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:33:49.318645 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:49.318669 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:49.319145 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:49.319564 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:49.321151 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:49.322539 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:49.324418 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:49.325301 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:49.325507 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:49.330015 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:49.330031 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:49.330051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:49.330106 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:49.334329 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:49.335375 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:49.335385 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:49.335395 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:49.335637 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:49.336147 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:49.336152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:49.336155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:49.337328 (+ 1173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:49.337790 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:49.338374 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 02:33:49.338893 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":440,"threads_started":1}
I20260620 02:33:49.340368 21223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:49.340591 21223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:49.343298 21223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36009 } }
I20260620 02:33:49.343886 21223 leader_election.cc:302] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01eb555031614a0caf4a3f8ab34b0cbb; no voters:
I20260620 02:33:49.344502 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:49.344628 21223 leader_election.cc:288] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:49.344739 21228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:49.347383 21223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:49.349238 21228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [term 1 LEADER]: Becoming Leader. State: Replica: 01eb555031614a0caf4a3f8ab34b0cbb, State: Running, Role: LEADER
I20260620 02:33:49.350265 21228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36009 } }
I20260620 02:33:49.353796 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01eb555031614a0caf4a3f8ab34b0cbb. Latest consensus state: current_term: 1 leader_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36009 } } }
I20260620 02:33:49.353833 21229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb555031614a0caf4a3f8ab34b0cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36009 } } }
I20260620 02:33:49.354242 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:49.354257 21229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:49.357559 21240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:49.366007 21240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:49.373800 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:49.349383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34159 (local address 127.0.0.1:36009)
0620 02:33:49.349596 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:49.349605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:49.350678 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 02:33:49.350942 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:49.350948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:49.351073 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:49.351189 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:49.356129 (+ 4940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:49.357527 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:49.358890 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:49.359350 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:49.359552 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:49.362724 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:49.362731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:49.362736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:49.362791 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:49.369304 (+ 6513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:49.370460 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:49.370466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:49.370470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:49.370611 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:49.371139 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:49.371143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:49.371146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:49.371595 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:49.372966 (+ 1371us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:49.373163 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 02:33:49.373406 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 02:33:49.377223 21250 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:33:49.379907 21240 catalog_manager.cc:1347] Generated new cluster ID: fd27d24f6d4a48ceb041dbdf90a8be9f
I20260620 02:33:49.380035 21240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:49.402257 21240 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:49.403713 21240 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:49.416101 21240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01eb555031614a0caf4a3f8ab34b0cbb: Generated new TSK 0
I20260620 02:33:49.417273 21240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:49.429272 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21156
I20260620 02:33:49.430696 21218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:49.525815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21156
2026-06-20T02:33:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:49 dist-test-slave-r706 krb5kdc[21255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:49 dist-test-slave-r706 krb5kdc[21255](info): set up 2 sockets
Jun 20 02:33:49 dist-test-slave-r706 krb5kdc[21255](info): commencing operation
krb5kdc: starting...
W20260620 02:33:50.665581 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:50 dist-test-slave-r706 krb5kdc[21255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:50.977883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:51.342262 21268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:51.342571 21268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:51.342741 21268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:51.350723 21268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:51.350841 21268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:51.350955 21268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:51.351064 21268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:51.351145 21268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:51.361591 21268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:51.362963 21268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:51.364333 21268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:51.379647 21274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:51.379741 21276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:51.380163 21273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:51.381426 21268 server_base.cc:1034] running on GCE node
I20260620 02:33:51.382333 21268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:51.383948 21268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:51.385248 21268 hybrid_clock.cc:648] HybridClock initialized: now 1781922831385207 us; error 68 us; skew 500 ppm
I20260620 02:33:51.385690 21268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:51 dist-test-slave-r706 krb5kdc[21255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:51.392310 21268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:51.393967 21268 webserver.cc:460] Webserver started at http://127.0.0.1:42567/ using document root <none> and password file <none>
I20260620 02:33:51.394695 21268 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:51.394806 21268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:51.395082 21268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:51.398459 21268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "f66716cfb4b941708f352e51bdaf9a1b"
format_stamp: "Formatted at 2026-06-20 02:33:51 on dist-test-slave-r706"
server_key: "3e9eb5b7679b4817327e774966055d19"
server_key_iv: "5bdd80a9cec8faf2fb242382617a33e5"
server_key_version: "encryptionkey@0"
I20260620 02:33:51.399261 21268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "f66716cfb4b941708f352e51bdaf9a1b"
format_stamp: "Formatted at 2026-06-20 02:33:51 on dist-test-slave-r706"
server_key: "3e9eb5b7679b4817327e774966055d19"
server_key_iv: "5bdd80a9cec8faf2fb242382617a33e5"
server_key_version: "encryptionkey@0"
I20260620 02:33:51.405948 21268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:33:51.410079 21283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:51.411686 21268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:51.411868 21268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "f66716cfb4b941708f352e51bdaf9a1b"
format_stamp: "Formatted at 2026-06-20 02:33:51 on dist-test-slave-r706"
server_key: "3e9eb5b7679b4817327e774966055d19"
server_key_iv: "5bdd80a9cec8faf2fb242382617a33e5"
server_key_version: "encryptionkey@0"
I20260620 02:33:51.412088 21268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:51.433225 21268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:51.437825 21268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:51.438074 21268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:51.458678 21268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45947
I20260620 02:33:51.458704 21334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45947 every 8 connection(s)
I20260620 02:33:51.460951 21268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:51.465826 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:51.469952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21268
I20260620 02:33:51.470160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:51.470803 388 external_mini_cluster.cc:1428] Setting key 14b49f9d4db1623d18545d634c2f7733
I20260620 02:33:51.479825 21335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b: Bootstrap starting.
I20260620 02:33:51.483739 21335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:51.484947 21335 log.cc:824] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:51.485191 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:51.472931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55529 (local address 127.0.0.1:45947)
0620 02:33:51.473811 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:51.473822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:51.473865 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:33:51.475556 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:51.475580 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:51.476037 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:51.476402 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:51.477144 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:51.478546 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:51.480513 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:51.481302 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:51.481454 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:51.482764 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:51.482771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:51.482786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:51.483341 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:51.483425 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:51.483452 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:51.483538 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:51.483801 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 02:33:51.484099 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":374,"threads_started":1}
I20260620 02:33:51.488058 21335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b: No bootstrap required, opened a new log
I20260620 02:33:51.488947 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:51.497131 21335 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66716cfb4b941708f352e51bdaf9a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260620 02:33:51.497663 21335 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:51.497794 21335 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66716cfb4b941708f352e51bdaf9a1b, State: Initialized, Role: FOLLOWER
I20260620 02:33:51.498533 21335 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66716cfb4b941708f352e51bdaf9a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260620 02:33:51.498844 21335 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:51.499010 21335 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:51.499192 21335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:51.500543 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:51.493526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43911 (local address 127.0.0.1:45947)
0620 02:33:51.493749 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:51.493755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:51.494036 (+ 281us) server_negotiation.cc:407] Connection header received
0620 02:33:51.494221 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:51.494225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:51.494293 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:51.494385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:51.495054 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:51.495850 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:51.497795 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:51.498335 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:51.498443 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:51.499737 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:51.499742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:51.499745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:51.499763 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:51.499802 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:51.499823 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:51.499922 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:51.500086 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 02:33:51.500170 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:33:51.502480 21335 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66716cfb4b941708f352e51bdaf9a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260620 02:33:51.503031 21335 leader_election.cc:302] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66716cfb4b941708f352e51bdaf9a1b; no voters:
I20260620 02:33:51.503648 21335 leader_election.cc:288] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:51.503834 21345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:51.504766 21345 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [term 1 LEADER]: Becoming Leader. State: Replica: f66716cfb4b941708f352e51bdaf9a1b, State: Running, Role: LEADER
I20260620 02:33:51.505414 21345 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66716cfb4b941708f352e51bdaf9a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260620 02:33:51.505990 21335 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:51.508234 21347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66716cfb4b941708f352e51bdaf9a1b. Latest consensus state: current_term: 1 leader_uuid: "f66716cfb4b941708f352e51bdaf9a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66716cfb4b941708f352e51bdaf9a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } } }
I20260620 02:33:51.508579 21347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:51.510113 21346 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66716cfb4b941708f352e51bdaf9a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66716cfb4b941708f352e51bdaf9a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } } }
I20260620 02:33:51.510874 21350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:51.510936 21346 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:51.518635 21350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:51.529316 21350 catalog_manager.cc:1347] Generated new cluster ID: fa6c90b93b184e3fada27399ffd30191
I20260620 02:33:51.529573 21350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:51.538870 21350 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:51.540807 21350 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:51.552518 21350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66716cfb4b941708f352e51bdaf9a1b: Generated new TSK 0
I20260620 02:33:51.553574 21350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:51.561164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21268
W20260620 02:33:51.562412 21330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55529
W20260620 02:33:51.563282 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21268: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:33:51.563472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21268
2026-06-20T02:33:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:51 dist-test-slave-r706 krb5kdc[21366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:51 dist-test-slave-r706 krb5kdc[21366](info): set up 2 sockets
Jun 20 02:33:51 dist-test-slave-r706 krb5kdc[21366](info): commencing operation
krb5kdc: starting...
W20260620 02:33:52.685592 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:52 dist-test-slave-r706 krb5kdc[21366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:53.005398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:53.348207 21379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:53.348573 21379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:53.348735 21379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:53.356319 21379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:53.356489 21379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:53.356594 21379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:53.356704 21379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:53.356791 21379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:53.367698 21379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:53.369158 21379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:53.370491 21379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:53.382505 21384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:53.382560 21385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:53.383080 21387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:53.384173 21379 server_base.cc:1034] running on GCE node
I20260620 02:33:53.385077 21379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:53.386584 21379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:53.387902 21379 hybrid_clock.cc:648] HybridClock initialized: now 1781922833387825 us; error 50 us; skew 500 ppm
I20260620 02:33:53.388343 21379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:53 dist-test-slave-r706 krb5kdc[21366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:53.391907 21379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:53.393721 21379 webserver.cc:460] Webserver started at http://127.0.0.1:38437/ using document root <none> and password file <none>
I20260620 02:33:53.394460 21379 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:53.394623 21379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:53.394968 21379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:53.398197 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "d54367ac79e44e0a8b0a560ae11793d5"
format_stamp: "Formatted at 2026-06-20 02:33:53 on dist-test-slave-r706"
server_key: "4754de13f0dbb4edeec33e8c1fa4020f"
server_key_iv: "f9e934219a89cfd88ddbf7d19847e642"
server_key_version: "encryptionkey@0"
I20260620 02:33:53.399030 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "d54367ac79e44e0a8b0a560ae11793d5"
format_stamp: "Formatted at 2026-06-20 02:33:53 on dist-test-slave-r706"
server_key: "4754de13f0dbb4edeec33e8c1fa4020f"
server_key_iv: "f9e934219a89cfd88ddbf7d19847e642"
server_key_version: "encryptionkey@0"
I20260620 02:33:53.404636 21379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:33:53.408541 21394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:53.410245 21379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 02:33:53.410480 21379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "d54367ac79e44e0a8b0a560ae11793d5"
format_stamp: "Formatted at 2026-06-20 02:33:53 on dist-test-slave-r706"
server_key: "4754de13f0dbb4edeec33e8c1fa4020f"
server_key_iv: "f9e934219a89cfd88ddbf7d19847e642"
server_key_version: "encryptionkey@0"
I20260620 02:33:53.410719 21379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:53.417066 21379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:53.421684 21379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:53.422166 21379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:53.443276 21379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36041
I20260620 02:33:53.443379 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36041 every 8 connection(s)
I20260620 02:33:53.444803 21379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:53.448724 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:53.454872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21379
I20260620 02:33:53.455076 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:53.455610 388 external_mini_cluster.cc:1428] Setting key 6d7ef439daf19ec7c4e914a6358e2825
I20260620 02:33:53.460310 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5: Bootstrap starting.
I20260620 02:33:53.463591 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:53.457539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41717 (local address 127.0.0.1:36041)
0620 02:33:53.458426 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:53.458437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:53.458478 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:33:53.460163 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:53.460194 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:53.460586 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:53.460978 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:53.462067 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:53.462072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:53.462084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:53.462099 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:53.462154 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:53.462183 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:53.462421 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:53.462557 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 02:33:53.462727 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":392,"threads_started":1}
I20260620 02:33:53.463832 21446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:53.465076 21446 log.cc:824] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:53.467682 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:53.468353 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5: No bootstrap required, opened a new log
I20260620 02:33:53.474972 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:53.472412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47649 (local address 127.0.0.1:36041)
0620 02:33:53.472618 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:53.472624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:53.472918 (+ 294us) server_negotiation.cc:407] Connection header received
0620 02:33:53.473119 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:53.473123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:53.473171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:53.473277 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:53.474244 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:53.474249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:53.474254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:53.474275 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:53.474316 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:53.474332 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:53.474437 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:53.474554 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 02:33:53.474607 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:33:53.477715 21446 raft_consensus.cc:348] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54367ac79e44e0a8b0a560ae11793d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260620 02:33:53.478199 21446 raft_consensus.cc:374] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:53.478330 21446 raft_consensus.cc:729] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54367ac79e44e0a8b0a560ae11793d5, State: Initialized, Role: FOLLOWER
I20260620 02:33:53.479043 21446 consensus_queue.cc:260] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54367ac79e44e0a8b0a560ae11793d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260620 02:33:53.479290 21446 raft_consensus.cc:388] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:53.479458 21446 raft_consensus.cc:482] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:53.479627 21446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:53.482007 21446 raft_consensus.cc:504] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54367ac79e44e0a8b0a560ae11793d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260620 02:33:53.482599 21446 leader_election.cc:302] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d54367ac79e44e0a8b0a560ae11793d5; no voters:
I20260620 02:33:53.483283 21446 leader_election.cc:288] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:53.483521 21456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:53.484318 21456 raft_consensus.cc:686] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [term 1 LEADER]: Becoming Leader. State: Replica: d54367ac79e44e0a8b0a560ae11793d5, State: Running, Role: LEADER
I20260620 02:33:53.484884 21456 consensus_queue.cc:237] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54367ac79e44e0a8b0a560ae11793d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260620 02:33:53.485543 21446 sys_catalog.cc:564] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:53.488466 21458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d54367ac79e44e0a8b0a560ae11793d5. Latest consensus state: current_term: 1 leader_uuid: "d54367ac79e44e0a8b0a560ae11793d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54367ac79e44e0a8b0a560ae11793d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20260620 02:33:53.489064 21458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:53.489630 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d54367ac79e44e0a8b0a560ae11793d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54367ac79e44e0a8b0a560ae11793d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20260620 02:33:53.489936 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:53.490757 21465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:53.496088 21465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:53.505689 21465 catalog_manager.cc:1347] Generated new cluster ID: b0b8f7d5df3242609ab9eac7005071df
I20260620 02:33:53.505820 21465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:53.543083 21465 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:53.543668 21465 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:53.567600 21465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d54367ac79e44e0a8b0a560ae11793d5: Generated new TSK 0
I20260620 02:33:53.568622 21465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:53.580554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21379
W20260620 02:33:53.581779 21441 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41717
W20260620 02:33:53.582553 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21379: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:33:53.582691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21379
2026-06-20T02:33:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:53 dist-test-slave-r706 krb5kdc[21477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:53 dist-test-slave-r706 krb5kdc[21477](info): set up 2 sockets
Jun 20 02:33:53 dist-test-slave-r706 krb5kdc[21477](info): commencing operation
krb5kdc: starting...
W20260620 02:33:54.725630 388 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:54 dist-test-slave-r706 krb5kdc[21477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:55.048393 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:55.422745 21490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:55.423058 21490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:55.423230 21490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:55.430768 21490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:55.430898 21490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:55.430981 21490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:55.431098 21490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:55.431169 21490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:55.441519 21490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:55.442934 21490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:55.444300 21490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:55.456741 21495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:55.456670 21496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:55.456768 21498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:55.457973 21490 server_base.cc:1034] running on GCE node
I20260620 02:33:55.458793 21490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:55.460281 21490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:55.461495 21490 hybrid_clock.cc:648] HybridClock initialized: now 1781922835461436 us; error 56 us; skew 500 ppm
I20260620 02:33:55.461877 21490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:55 dist-test-slave-r706 krb5kdc[21477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:55.465404 21490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:55.466928 21490 webserver.cc:460] Webserver started at http://127.0.0.1:37933/ using document root <none> and password file <none>
I20260620 02:33:55.467597 21490 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:55.467736 21490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:55.468073 21490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:55.471244 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07"
format_stamp: "Formatted at 2026-06-20 02:33:55 on dist-test-slave-r706"
server_key: "fcd975a48d3bce311f7a6311d12e24ed"
server_key_iv: "dd8b46ec7242ffe505a60804a2468928"
server_key_version: "encryptionkey@0"
I20260620 02:33:55.472052 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07"
format_stamp: "Formatted at 2026-06-20 02:33:55 on dist-test-slave-r706"
server_key: "fcd975a48d3bce311f7a6311d12e24ed"
server_key_iv: "dd8b46ec7242ffe505a60804a2468928"
server_key_version: "encryptionkey@0"
I20260620 02:33:55.477763 21490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 02:33:55.481987 21505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:55.483449 21490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:55.483667 21490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07"
format_stamp: "Formatted at 2026-06-20 02:33:55 on dist-test-slave-r706"
server_key: "fcd975a48d3bce311f7a6311d12e24ed"
server_key_iv: "dd8b46ec7242ffe505a60804a2468928"
server_key_version: "encryptionkey@0"
I20260620 02:33:55.483892 21490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:55.499421 21490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:55.503933 21490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:55.504251 21490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:55.525522 21490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40105
I20260620 02:33:55.525676 21556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40105 every 8 connection(s)
I20260620 02:33:55.527793 21490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:55.529268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21490
I20260620 02:33:55.529465 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:55.530009 388 external_mini_cluster.cc:1428] Setting key d6f35f8ea711e41b3550493bfb040ec7
I20260620 02:33:55.535236 21557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:33:55 dist-test-slave-r706 krb5kdc[21477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:33:55.549741 21557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07: Bootstrap starting.
I20260620 02:33:55.553357 21557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:55.554625 21557 log.cc:824] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:55.557331 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:55.532456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:58927 (local address 127.0.0.1:40105)
0620 02:33:55.534198 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:55.534210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:55.534252 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:33:55.536043 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:55.536066 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:55.536463 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:55.536805 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:55.538932 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:55.540547 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:55.542522 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:55.543275 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:55.543435 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:55.548880 (+ 5445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:55.548888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:55.548906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:55.548946 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:55.552261 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:55.553380 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:55.553391 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:55.553403 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:55.553680 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:55.554274 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:55.554278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:55.554282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:55.554926 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:55.555281 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:55.555866 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 02:33:55.556272 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1212,"threads_started":1}
I20260620 02:33:55.558352 21557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07: No bootstrap required, opened a new log
I20260620 02:33:55.561704 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:55.567950 21557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40105 } }
I20260620 02:33:55.568400 21557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:55.568535 21557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202ef05b4e4a4f3e9d0ef96c9217cd07, State: Initialized, Role: FOLLOWER
I20260620 02:33:55.569298 21557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40105 } }
I20260620 02:33:55.569571 21557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:55.569726 21557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:55.569923 21557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:55.572258 21557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40105 } }
I20260620 02:33:55.572779 21557 leader_election.cc:302] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 202ef05b4e4a4f3e9d0ef96c9217cd07; no voters:
I20260620 02:33:55.573462 21557 leader_election.cc:288] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:55.573643 21567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:55.574896 21567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [term 1 LEADER]: Becoming Leader. State: Replica: 202ef05b4e4a4f3e9d0ef96c9217cd07, State: Running, Role: LEADER
I20260620 02:33:55.575785 21557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:55.575619 21567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40105 } }
I20260620 02:33:55.579733 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202ef05b4e4a4f3e9d0ef96c9217cd07. Latest consensus state: current_term: 1 leader_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40105 } } }
I20260620 02:33:55.580116 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:55.580570 21568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202ef05b4e4a4f3e9d0ef96c9217cd07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40105 } } }
I20260620 02:33:55.580878 21568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:55.584528 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:55.566164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:35699 (local address 127.0.0.1:40105)
0620 02:33:55.566385 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:55.566392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:55.566724 (+ 332us) server_negotiation.cc:407] Connection header received
0620 02:33:55.566887 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:55.566894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:55.566984 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:55.567113 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:55.568576 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:55.569562 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:55.570944 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:55.571432 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:55.571575 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:55.574953 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:55.574962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:55.574969 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:33:55.575025 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:55.579097 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:55.580344 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:55.580350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:55.580357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:55.580574 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:33:55.581337 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:33:55.581345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:33:55.581351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:33:55.581876 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:55.583451 (+ 1575us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:55.583711 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 02:33:55.583965 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:33:55.594302 21575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:55.600808 21575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:55.611405 21575 catalog_manager.cc:1347] Generated new cluster ID: 85702c98bbf04333b01610dcc06203a0
I20260620 02:33:55.611553 21575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:55.624557 21575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:55.626008 21575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:55.636861 21575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 202ef05b4e4a4f3e9d0ef96c9217cd07: Generated new TSK 0
I20260620 02:33:55.637811 21575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:55.644098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21490
I20260620 02:33:55.645653 21552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:33:55.738966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21490
W20260620 02:33:55.758195 21475 connection.cc:535] client connection to 127.0.0.1:40105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40105): Connection reset by peer (error 104)
2026-06-20T02:33:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2158 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:55 dist-test-slave-r706 krb5kdc[21589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:55 dist-test-slave-r706 krb5kdc[21589](info): set up 2 sockets
Jun 20 02:33:55 dist-test-slave-r706 krb5kdc[21589](info): commencing operation
krb5kdc: starting...
W20260620 02:33:56.937680 388 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:57 dist-test-slave-r706 krb5kdc[21589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:57.256232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:57.607196 21602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:57.607520 21602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:57.607707 21602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:57.615272 21602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:57.615406 21602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:57.615454 21602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:57.615511 21602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:57.615613 21602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:57.626590 21602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:57.628019 21602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:57.629438 21602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:57.641868 21607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:57.641834 21608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:57.642366 21610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:57.642758 21602 server_base.cc:1034] running on GCE node
I20260620 02:33:57.643596 21602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:57.645206 21602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:57.646482 21602 hybrid_clock.cc:648] HybridClock initialized: now 1781922837646442 us; error 38 us; skew 500 ppm
I20260620 02:33:57.646899 21602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:57 dist-test-slave-r706 krb5kdc[21589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:57.655107 21602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:57.656898 21602 webserver.cc:460] Webserver started at http://127.0.0.1:34717/ using document root <none> and password file <none>
I20260620 02:33:57.657668 21602 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:57.657833 21602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:57.658180 21602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:57.661423 21602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "c320fb8dde65468f91ab6aca6210e9aa"
format_stamp: "Formatted at 2026-06-20 02:33:57 on dist-test-slave-r706"
server_key: "3b40ae6dc2dca3834f21bf97e1d2f8f5"
server_key_iv: "5a0a1efc98146d360086d487dac9486c"
server_key_version: "encryptionkey@0"
I20260620 02:33:57.662261 21602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "c320fb8dde65468f91ab6aca6210e9aa"
format_stamp: "Formatted at 2026-06-20 02:33:57 on dist-test-slave-r706"
server_key: "3b40ae6dc2dca3834f21bf97e1d2f8f5"
server_key_iv: "5a0a1efc98146d360086d487dac9486c"
server_key_version: "encryptionkey@0"
I20260620 02:33:57.667786 21602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:33:57.671471 21617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:57.672907 21602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:33:57.673158 21602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "c320fb8dde65468f91ab6aca6210e9aa"
format_stamp: "Formatted at 2026-06-20 02:33:57 on dist-test-slave-r706"
server_key: "3b40ae6dc2dca3834f21bf97e1d2f8f5"
server_key_iv: "5a0a1efc98146d360086d487dac9486c"
server_key_version: "encryptionkey@0"
I20260620 02:33:57.673377 21602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:57.695698 21602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:57.700531 21602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:57.700878 21602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:57.720608 21602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43217
I20260620 02:33:57.720647 21668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43217 every 8 connection(s)
I20260620 02:33:57.722096 21602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:57.725903 21669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:57.727586 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21602
I20260620 02:33:57.727799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:57.728343 388 external_mini_cluster.cc:1428] Setting key 116a8447e8f689a9650b95bdcbf8d2df
I20260620 02:33:57.740926 21669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa: Bootstrap starting.
I20260620 02:33:57.742663 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:57.730299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:50475 (local address 127.0.0.1:43217)
0620 02:33:57.731142 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:57.731159 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:57.731205 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:33:57.732918 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:57.732944 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:57.733489 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:57.733848 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:57.734481 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:57.735786 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:57.737913 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:57.738680 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:57.738834 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:57.740159 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:57.740168 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:57.740183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:57.740759 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:57.740837 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:57.740862 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:57.740985 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:57.741504 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 02:33:57.741687 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":397,"threads_started":1}
I20260620 02:33:57.744297 21669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:57.745549 21669 log.cc:824] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:57.746708 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:57.748979 21669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa: No bootstrap required, opened a new log
I20260620 02:33:57.758996 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:57.751929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:53937 (local address 127.0.0.1:43217)
0620 02:33:57.752161 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:57.752167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:57.752461 (+ 294us) server_negotiation.cc:407] Connection header received
0620 02:33:57.752659 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:57.752664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:57.752729 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:57.752816 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:57.753506 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:57.754321 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:57.756248 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:33:57.756739 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:33:57.756851 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:33:57.758181 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:57.758186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:57.758190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:57.758209 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:57.758271 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:57.758287 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:57.758375 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:57.758591 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 02:33:57.758638 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 02:33:57.759183 21669 raft_consensus.cc:348] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c320fb8dde65468f91ab6aca6210e9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260620 02:33:57.759701 21669 raft_consensus.cc:374] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:57.759824 21669 raft_consensus.cc:729] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c320fb8dde65468f91ab6aca6210e9aa, State: Initialized, Role: FOLLOWER
I20260620 02:33:57.760567 21669 consensus_queue.cc:260] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c320fb8dde65468f91ab6aca6210e9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260620 02:33:57.760898 21669 raft_consensus.cc:388] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:57.761065 21669 raft_consensus.cc:482] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:57.761224 21669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:57.765247 21669 raft_consensus.cc:504] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c320fb8dde65468f91ab6aca6210e9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260620 02:33:57.766148 21669 leader_election.cc:302] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c320fb8dde65468f91ab6aca6210e9aa; no voters:
I20260620 02:33:57.767071 21669 leader_election.cc:288] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:57.767248 21679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:57.768918 21679 raft_consensus.cc:686] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [term 1 LEADER]: Becoming Leader. State: Replica: c320fb8dde65468f91ab6aca6210e9aa, State: Running, Role: LEADER
I20260620 02:33:57.770004 21679 consensus_queue.cc:237] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c320fb8dde65468f91ab6aca6210e9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260620 02:33:57.771116 21669 sys_catalog.cc:564] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:57.780751 21680 sys_catalog.cc:455] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c320fb8dde65468f91ab6aca6210e9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c320fb8dde65468f91ab6aca6210e9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } } }
I20260620 02:33:57.782546 21680 sys_catalog.cc:458] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:57.782600 21681 sys_catalog.cc:455] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c320fb8dde65468f91ab6aca6210e9aa. Latest consensus state: current_term: 1 leader_uuid: "c320fb8dde65468f91ab6aca6210e9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c320fb8dde65468f91ab6aca6210e9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } } }
I20260620 02:33:57.787663 21681 sys_catalog.cc:458] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:57.789427 21688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:57.799023 21688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:57.817477 21688 catalog_manager.cc:1347] Generated new cluster ID: b182e0ed0b6c47c9bc5531403299872c
I20260620 02:33:57.817730 21688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:57.854096 21688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:57.857777 21688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:57.876451 21688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c320fb8dde65468f91ab6aca6210e9aa: Generated new TSK 0
I20260620 02:33:57.878149 21688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:57.898473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21602
W20260620 02:33:57.901057 21664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:50475
W20260620 02:33:57.902374 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:33:57.902668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21602
W20260620 02:33:57.935657 21587 connection.cc:535] client connection to 127.0.0.1:43217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43217): Connection reset by peer (error 104)
2026-06-20T02:33:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:33:58 dist-test-slave-r706 krb5kdc[21700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:33:58 dist-test-slave-r706 krb5kdc[21700](info): set up 2 sockets
Jun 20 02:33:58 dist-test-slave-r706 krb5kdc[21700](info): commencing operation
krb5kdc: starting...
W20260620 02:33:59.066035 388 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:33:59 dist-test-slave-r706 krb5kdc[21700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:33:59.396085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:33:59.758306 21713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:33:59.758633 21713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:33:59.758786 21713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:33:59.766279 21713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:33:59.766413 21713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:33:59.766510 21713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:33:59.766628 21713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:33:59.766712 21713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:33:59.777371 21713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:33:59.778882 21713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:33:59.780246 21713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:33:59.792979 21718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:59.793081 21719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:33:59.793057 21721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:33:59.793565 21713 server_base.cc:1034] running on GCE node
I20260620 02:33:59.794708 21713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:33:59.796249 21713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:33:59.797463 21713 hybrid_clock.cc:648] HybridClock initialized: now 1781922839797421 us; error 32 us; skew 500 ppm
I20260620 02:33:59.797860 21713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:33:59 dist-test-slave-r706 krb5kdc[21700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:33:59.801246 21713 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:33:59.802835 21713 webserver.cc:460] Webserver started at http://127.0.0.1:42649/ using document root <none> and password file <none>
I20260620 02:33:59.803578 21713 fs_manager.cc:359] Metadata directory not provided
I20260620 02:33:59.803732 21713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:33:59.804056 21713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:33:59.807256 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "b08eeb699afc481da3e0ef5af8218862"
format_stamp: "Formatted at 2026-06-20 02:33:59 on dist-test-slave-r706"
server_key: "8120288ad68dbc031442e0c4c24f9273"
server_key_iv: "602560d07ee4747a9dbfa04488e10598"
server_key_version: "encryptionkey@0"
I20260620 02:33:59.808080 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "b08eeb699afc481da3e0ef5af8218862"
format_stamp: "Formatted at 2026-06-20 02:33:59 on dist-test-slave-r706"
server_key: "8120288ad68dbc031442e0c4c24f9273"
server_key_iv: "602560d07ee4747a9dbfa04488e10598"
server_key_version: "encryptionkey@0"
I20260620 02:33:59.813825 21713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:33:59.817857 21728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:33:59.819506 21713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:33:59.819753 21713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "b08eeb699afc481da3e0ef5af8218862"
format_stamp: "Formatted at 2026-06-20 02:33:59 on dist-test-slave-r706"
server_key: "8120288ad68dbc031442e0c4c24f9273"
server_key_iv: "602560d07ee4747a9dbfa04488e10598"
server_key_version: "encryptionkey@0"
I20260620 02:33:59.820009 21713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:33:59.826303 21713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:33:59.830814 21713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:33:59.831163 21713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:33:59.850844 21713 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37703
I20260620 02:33:59.850904 21779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37703 every 8 connection(s)
I20260620 02:33:59.852355 21713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:33:59.856417 21780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:33:59.860154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21713
I20260620 02:33:59.860376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:33:59.860927 388 external_mini_cluster.cc:1428] Setting key ab0a02a0fca796293e68caeee865b859
I20260620 02:33:59.870180 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:59.863647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:52517 (local address 127.0.0.1:37703)
0620 02:33:59.864534 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:59.864546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:59.864589 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:33:59.866435 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:59.866462 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:59.866883 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:59.867275 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:59.868479 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:59.868486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:59.868500 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:59.868518 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:59.868579 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:59.868599 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:59.868814 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:59.868916 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 02:33:59.869180 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":396,"threads_started":1}
I20260620 02:33:59.870951 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862: Bootstrap starting.
I20260620 02:33:59.874166 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:33:59.874522 21780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862: Neither blocks nor log segments found. Creating new log.
I20260620 02:33:59.875831 21780 log.cc:824] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862: Log is configured to *not* fsync() on all Append() calls
I20260620 02:33:59.879617 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862: No bootstrap required, opened a new log
I20260620 02:33:59.882229 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:33:59.879467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:46883 (local address 127.0.0.1:37703)
0620 02:33:59.879697 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:33:59.879705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:33:59.880187 (+ 482us) server_negotiation.cc:407] Connection header received
0620 02:33:59.880340 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:33:59.880345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:33:59.880410 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:33:59.880535 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:33:59.881534 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:33:59.881541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:33:59.881546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:33:59.881566 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:33:59.881632 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:33:59.881653 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:33:59.881792 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:33:59.881852 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 02:33:59.881919 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:33:59.889144 21780 raft_consensus.cc:348] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08eeb699afc481da3e0ef5af8218862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260620 02:33:59.889643 21780 raft_consensus.cc:374] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:33:59.889760 21780 raft_consensus.cc:729] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08eeb699afc481da3e0ef5af8218862, State: Initialized, Role: FOLLOWER
I20260620 02:33:59.890465 21780 consensus_queue.cc:260] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08eeb699afc481da3e0ef5af8218862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260620 02:33:59.890770 21780 raft_consensus.cc:388] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:33:59.890941 21780 raft_consensus.cc:482] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:33:59.891165 21780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:33:59.893817 21780 raft_consensus.cc:504] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08eeb699afc481da3e0ef5af8218862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260620 02:33:59.894343 21780 leader_election.cc:302] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b08eeb699afc481da3e0ef5af8218862; no voters:
I20260620 02:33:59.894968 21780 leader_election.cc:288] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:33:59.895143 21790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:33:59.895840 21790 raft_consensus.cc:686] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [term 1 LEADER]: Becoming Leader. State: Replica: b08eeb699afc481da3e0ef5af8218862, State: Running, Role: LEADER
I20260620 02:33:59.896464 21790 consensus_queue.cc:237] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08eeb699afc481da3e0ef5af8218862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } }
I20260620 02:33:59.897387 21780 sys_catalog.cc:564] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:33:59.899879 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b08eeb699afc481da3e0ef5af8218862. Latest consensus state: current_term: 1 leader_uuid: "b08eeb699afc481da3e0ef5af8218862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08eeb699afc481da3e0ef5af8218862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } } }
I20260620 02:33:59.900446 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:59.899904 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b08eeb699afc481da3e0ef5af8218862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08eeb699afc481da3e0ef5af8218862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37703 } } }
I20260620 02:33:59.900585 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862 [sys.catalog]: This master's current role is: LEADER
I20260620 02:33:59.903631 21799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:33:59.908568 21799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:33:59.918896 21799 catalog_manager.cc:1347] Generated new cluster ID: 3d149f785213452d8b47fdcda9f1dbdb
I20260620 02:33:59.919099 21799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:33:59.940363 21799 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:33:59.940912 21799 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:33:59.949903 21799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b08eeb699afc481da3e0ef5af8218862: Generated new TSK 0
I20260620 02:33:59.950838 21799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:33:59.960031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21713
W20260620 02:33:59.961377 21775 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:52517
W20260620 02:33:59.962124 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21713: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:33:59.962296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21713
2026-06-20T02:33:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:00 dist-test-slave-r706 krb5kdc[21811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:00 dist-test-slave-r706 krb5kdc[21811](info): set up 2 sockets
Jun 20 02:34:00 dist-test-slave-r706 krb5kdc[21811](info): commencing operation
krb5kdc: starting...
W20260620 02:34:01.127808 388 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.007s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:01 dist-test-slave-r706 krb5kdc[21811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:34:01.436326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:01.782162 21824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:01.782611 21824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:01.782886 21824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:01.790555 21824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:01.790743 21824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:01.790861 21824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:01.790969 21824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:01.791059 21824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:01.802067 21824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:01.803584 21824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:01.805019 21824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:01.818274 21829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:01.818325 21832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:01.818298 21830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:01.819177 21824 server_base.cc:1034] running on GCE node
I20260620 02:34:01.820055 21824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:01.821681 21824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:01.822937 21824 hybrid_clock.cc:648] HybridClock initialized: now 1781922841822891 us; error 43 us; skew 500 ppm
I20260620 02:34:01.823323 21824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:01 dist-test-slave-r706 krb5kdc[21811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:01.826797 21824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:34:01.828539 21824 webserver.cc:460] Webserver started at http://127.0.0.1:39309/ using document root <none> and password file <none>
I20260620 02:34:01.829352 21824 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:01.829515 21824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:01.829859 21824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:01.833150 21824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "60c699b18cf24e90a78996c5616fbe3d"
format_stamp: "Formatted at 2026-06-20 02:34:01 on dist-test-slave-r706"
server_key: "0cb06100c6201e869cb28606e35d39aa"
server_key_iv: "3323608c750370dd81fdebc851b849bf"
server_key_version: "encryptionkey@0"
I20260620 02:34:01.834188 21824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "60c699b18cf24e90a78996c5616fbe3d"
format_stamp: "Formatted at 2026-06-20 02:34:01 on dist-test-slave-r706"
server_key: "0cb06100c6201e869cb28606e35d39aa"
server_key_iv: "3323608c750370dd81fdebc851b849bf"
server_key_version: "encryptionkey@0"
I20260620 02:34:01.840021 21824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 02:34:01.844089 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:01.845578 21824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:34:01.845790 21824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "60c699b18cf24e90a78996c5616fbe3d"
format_stamp: "Formatted at 2026-06-20 02:34:01 on dist-test-slave-r706"
server_key: "0cb06100c6201e869cb28606e35d39aa"
server_key_iv: "3323608c750370dd81fdebc851b849bf"
server_key_version: "encryptionkey@0"
I20260620 02:34:01.846019 21824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:01.866822 21824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:01.871565 21824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:01.871901 21824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:01.891626 21824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38595
I20260620 02:34:01.891670 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38595 every 8 connection(s)
I20260620 02:34:01.893126 21824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:01.896230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21824
I20260620 02:34:01.896448 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:01.896957 388 external_mini_cluster.cc:1428] Setting key 269a4b2aec0a34acb698ac2cc9771380
I20260620 02:34:01.897176 21891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:01.912632 21891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d: Bootstrap starting.
Jun 20 02:34:01 dist-test-slave-r706 krb5kdc[21811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:34:01.916014 21891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:01.917248 21891 log.cc:824] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:01.920197 21891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d: No bootstrap required, opened a new log
I20260620 02:34:01.922041 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:01.899004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:39743 (local address 127.0.0.1:38595)
0620 02:34:01.899875 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:01.899886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:01.899956 (+ 70us) server_negotiation.cc:407] Connection header received
0620 02:34:01.902076 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:01.902098 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:01.902535 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:01.902889 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:01.904831 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:01.906285 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:01.908112 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:01.908883 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:01.909124 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:01.914161 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:01.914170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:01.914186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:01.914231 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:01.917609 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:01.918499 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:01.918506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:01.918513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:01.918701 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:01.919185 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:01.919192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:01.919201 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:01.919729 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:01.920076 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:01.920613 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 02:34:01.920969 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":359,"threads_started":1}
I20260620 02:34:01.926384 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:01.928812 21891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c699b18cf24e90a78996c5616fbe3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38595 } }
I20260620 02:34:01.929351 21891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:01.929490 21891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c699b18cf24e90a78996c5616fbe3d, State: Initialized, Role: FOLLOWER
I20260620 02:34:01.930159 21891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c699b18cf24e90a78996c5616fbe3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38595 } }
I20260620 02:34:01.930411 21891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:01.930577 21891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:01.930811 21891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:01.934352 21891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c699b18cf24e90a78996c5616fbe3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38595 } }
I20260620 02:34:01.934895 21891 leader_election.cc:302] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60c699b18cf24e90a78996c5616fbe3d; no voters:
I20260620 02:34:01.935575 21891 leader_election.cc:288] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:01.935702 21901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:01.938380 21891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:01.941329 21901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [term 1 LEADER]: Becoming Leader. State: Replica: 60c699b18cf24e90a78996c5616fbe3d, State: Running, Role: LEADER
I20260620 02:34:01.942195 21901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c699b18cf24e90a78996c5616fbe3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38595 } }
I20260620 02:34:01.946166 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60c699b18cf24e90a78996c5616fbe3d. Latest consensus state: current_term: 1 leader_uuid: "60c699b18cf24e90a78996c5616fbe3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c699b18cf24e90a78996c5616fbe3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38595 } } }
I20260620 02:34:01.947049 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:01.946918 21902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60c699b18cf24e90a78996c5616fbe3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c699b18cf24e90a78996c5616fbe3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38595 } } }
I20260620 02:34:01.947264 21902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:01.948104 21908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:01.952165 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:01.931073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:42285 (local address 127.0.0.1:38595)
0620 02:34:01.931255 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:01.931260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:01.933567 (+ 2307us) server_negotiation.cc:407] Connection header received
0620 02:34:01.933780 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:01.933786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:01.933861 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:01.933972 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:01.935598 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:01.936872 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:01.938294 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:01.938890 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:01.939021 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:01.942079 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:01.942084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:01.942088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:01.942122 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:01.946036 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:01.947706 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:01.947713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:01.947719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:01.947892 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:01.948652 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:01.948660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:01.948666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:01.949265 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:01.951102 (+ 1837us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:01.951334 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 02:34:01.951615 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:34:01.964648 21908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:01.975651 21908 catalog_manager.cc:1347] Generated new cluster ID: 89379cd868f04053bb2b03d7f9de9729
I20260620 02:34:01.975790 21908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:01.988873 21908 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:01.990285 21908 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:02.000389 21908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60c699b18cf24e90a78996c5616fbe3d: Generated new TSK 0
I20260620 02:34:02.001364 21908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:02.008245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21824
I20260620 02:34:02.010360 21886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:02.104156 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21824
W20260620 02:34:02.123229 21809 connection.cc:535] client connection to 127.0.0.1:38595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38595): Connection reset by peer (error 104)
2026-06-20T02:34:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2146 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:02 dist-test-slave-r706 krb5kdc[21923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:02 dist-test-slave-r706 krb5kdc[21923](info): set up 2 sockets
Jun 20 02:34:02 dist-test-slave-r706 krb5kdc[21923](info): commencing operation
krb5kdc: starting...
W20260620 02:34:03.253957 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:03 dist-test-slave-r706 krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:34:03.556826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:03.918632 21936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:03.918969 21936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:03.919137 21936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:03.926962 21936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:03.927129 21936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:03.927246 21936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:03.927367 21936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:03.927421 21936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:03.938436 21936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:03.939843 21936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:03.941275 21936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:03.954736 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:03.954723 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:03.954684 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:03.956262 21936 server_base.cc:1034] running on GCE node
I20260620 02:34:03.957269 21936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:03.958837 21936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:03.960121 21936 hybrid_clock.cc:648] HybridClock initialized: now 1781922843960057 us; error 53 us; skew 500 ppm
I20260620 02:34:03.960510 21936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:03 dist-test-slave-r706 krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:03.964129 21936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:34:03.965932 21936 webserver.cc:460] Webserver started at http://127.0.0.1:46607/ using document root <none> and password file <none>
I20260620 02:34:03.966790 21936 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:03.966951 21936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:03.967305 21936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:03.970649 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "4c145df92d304761981a9a116ee10b72"
format_stamp: "Formatted at 2026-06-20 02:34:03 on dist-test-slave-r706"
server_key: "56de180f9b39373bacd54019531c4c98"
server_key_iv: "d63850b0434412675f529cbd79843262"
server_key_version: "encryptionkey@0"
I20260620 02:34:03.971483 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c145df92d304761981a9a116ee10b72"
format_stamp: "Formatted at 2026-06-20 02:34:03 on dist-test-slave-r706"
server_key: "56de180f9b39373bacd54019531c4c98"
server_key_iv: "d63850b0434412675f529cbd79843262"
server_key_version: "encryptionkey@0"
I20260620 02:34:03.977658 21936 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:34:03.982165 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:03.983769 21936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:34:03.984180 21936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "4c145df92d304761981a9a116ee10b72"
format_stamp: "Formatted at 2026-06-20 02:34:03 on dist-test-slave-r706"
server_key: "56de180f9b39373bacd54019531c4c98"
server_key_iv: "d63850b0434412675f529cbd79843262"
server_key_version: "encryptionkey@0"
I20260620 02:34:03.984568 21936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:04.027303 21936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:04.037163 21936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:04.037746 21936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:04.078531 21936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44893
I20260620 02:34:04.078662 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44893 every 8 connection(s)
I20260620 02:34:04.081336 21936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:04.088707 22003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:04.090893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 21936
I20260620 02:34:04.091202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:04.091981 388 external_mini_cluster.cc:1428] Setting key 7cf43225b1131d1186ff6a33793666b2
I20260620 02:34:04.114398 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72: Bootstrap starting.
I20260620 02:34:04.117677 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:04.095685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:36265 (local address 127.0.0.1:44893)
0620 02:34:04.097362 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:04.097380 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:04.097467 (+ 87us) server_negotiation.cc:407] Connection header received
0620 02:34:04.100859 (+ 3392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:04.100912 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:04.101936 (+ 1024us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:04.102641 (+ 705us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:04.104105 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:04.106406 (+ 2301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:04.110147 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:04.111735 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:04.112007 (+ 272us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:04.113542 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:04.113558 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:04.113586 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:04.114481 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:04.114633 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:04.114780 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:04.115008 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:04.115323 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 02:34:04.115540 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":756,"threads_started":1}
I20260620 02:34:04.118919 22003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:04.120410 22003 log.cc:824] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:04.123718 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:04.125386 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72: No bootstrap required, opened a new log
I20260620 02:34:04.145383 22003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c145df92d304761981a9a116ee10b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44893 } }
I20260620 02:34:04.146531 22003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:04.146816 22003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c145df92d304761981a9a116ee10b72, State: Initialized, Role: FOLLOWER
I20260620 02:34:04.148469 22003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c145df92d304761981a9a116ee10b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44893 } }
I20260620 02:34:04.149258 22003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:04.149638 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:04.150015 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:04.154723 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:04.132484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:38179 (local address 127.0.0.1:44893)
0620 02:34:04.133150 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:04.133167 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:04.138522 (+ 5355us) server_negotiation.cc:407] Connection header received
0620 02:34:04.138854 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:04.138867 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:04.139101 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:04.139414 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:04.140477 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:04.142795 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:04.145777 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:04.149155 (+ 3378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:04.149445 (+ 290us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:04.152146 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:04.152166 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:04.152180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:04.152229 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:04.152379 (+ 150us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:04.152449 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:04.152845 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:04.153441 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 02:34:04.153628 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260620 02:34:04.159145 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c145df92d304761981a9a116ee10b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44893 } }
I20260620 02:34:04.160349 22003 leader_election.cc:302] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c145df92d304761981a9a116ee10b72; no voters:
I20260620 02:34:04.161614 22003 leader_election.cc:288] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:04.161850 22013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:04.166112 22003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:04.170315 22013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [term 1 LEADER]: Becoming Leader. State: Replica: 4c145df92d304761981a9a116ee10b72, State: Running, Role: LEADER
I20260620 02:34:04.172415 22013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c145df92d304761981a9a116ee10b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44893 } }
I20260620 02:34:04.179502 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c145df92d304761981a9a116ee10b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c145df92d304761981a9a116ee10b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44893 } } }
I20260620 02:34:04.180403 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:04.179636 22015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c145df92d304761981a9a116ee10b72. Latest consensus state: current_term: 1 leader_uuid: "4c145df92d304761981a9a116ee10b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c145df92d304761981a9a116ee10b72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44893 } } }
I20260620 02:34:04.182082 22015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:04.186190 22024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:04.192188 22024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:04.202939 22024 catalog_manager.cc:1347] Generated new cluster ID: 0964c32d4e3f4d3687de72f010f494c7
I20260620 02:34:04.203090 22024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:04.230330 22024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:04.232668 22024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:04.246767 22024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c145df92d304761981a9a116ee10b72: Generated new TSK 0
I20260620 02:34:04.247802 22024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:04.261332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21936
W20260620 02:34:04.262838 21998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:36265
W20260620 02:34:04.263777 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 21936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:34:04.263962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 21936
W20260620 02:34:04.284507 21921 connection.cc:535] client connection to 127.0.0.1:44893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44893): Connection reset by peer (error 104)
2026-06-20T02:34:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2157 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:04 dist-test-slave-r706 krb5kdc[22034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:04 dist-test-slave-r706 krb5kdc[22034](info): set up 2 sockets
Jun 20 02:34:04 dist-test-slave-r706 krb5kdc[22034](info): commencing operation
krb5kdc: starting...
W20260620 02:34:05.417529 388 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.005s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:05 dist-test-slave-r706 krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:34:05.724289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:06.088912 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:06.089372 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:06.089557 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:06.097412 22047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:06.097546 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:06.097625 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:06.097723 22047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:06.097816 22047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:06.109516 22047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:06.110965 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:06.112358 22047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:06.126180 22052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:06.126258 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:06.126233 22053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:06.126894 22047 server_base.cc:1034] running on GCE node
I20260620 02:34:06.128193 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:06.129827 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:06.131086 22047 hybrid_clock.cc:648] HybridClock initialized: now 1781922846131030 us; error 54 us; skew 500 ppm
I20260620 02:34:06.131446 22047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:06 dist-test-slave-r706 krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:06.135910 22047 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:34:06.137781 22047 webserver.cc:460] Webserver started at http://127.0.0.1:33795/ using document root <none> and password file <none>
I20260620 02:34:06.138568 22047 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:06.138721 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:06.139097 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:06.142769 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "200c01e5d0744708a0736f1c6f4b5acf"
format_stamp: "Formatted at 2026-06-20 02:34:06 on dist-test-slave-r706"
server_key: "1e33d6649fa85aaa20891c6e9177e973"
server_key_iv: "fe43a85cdbf4611cb1ddde1bf4aa87d6"
server_key_version: "encryptionkey@0"
I20260620 02:34:06.143641 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "200c01e5d0744708a0736f1c6f4b5acf"
format_stamp: "Formatted at 2026-06-20 02:34:06 on dist-test-slave-r706"
server_key: "1e33d6649fa85aaa20891c6e9177e973"
server_key_iv: "fe43a85cdbf4611cb1ddde1bf4aa87d6"
server_key_version: "encryptionkey@0"
I20260620 02:34:06.150175 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:34:06.154573 22062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:06.156685 22047 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260620 02:34:06.156955 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "200c01e5d0744708a0736f1c6f4b5acf"
format_stamp: "Formatted at 2026-06-20 02:34:06 on dist-test-slave-r706"
server_key: "1e33d6649fa85aaa20891c6e9177e973"
server_key_iv: "fe43a85cdbf4611cb1ddde1bf4aa87d6"
server_key_version: "encryptionkey@0"
I20260620 02:34:06.157261 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:06.164736 22047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:06.169629 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:06.169965 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:06.191815 22047 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42475
I20260620 02:34:06.191859 22113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42475 every 8 connection(s)
I20260620 02:34:06.193499 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:06.195676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22047
I20260620 02:34:06.195864 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:06.196380 388 external_mini_cluster.cc:1428] Setting key 3419fc4eb58270800aa33644bb5dc359
I20260620 02:34:06.197906 22114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:06.204502 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:06.198339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:50679 (local address 127.0.0.1:42475)
0620 02:34:06.199194 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:06.199205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:06.199247 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:34:06.200833 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:06.200860 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:06.201308 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:06.201680 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:06.202829 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:06.202835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:06.202847 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:06.202862 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:06.202944 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:06.202965 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:06.203178 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:06.203283 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 02:34:06.203507 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":384,"threads_started":1}
I20260620 02:34:06.208400 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:06.211129 22114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf: Bootstrap starting.
I20260620 02:34:06.215557 22114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:06.215775 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:06.212983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:58091 (local address 127.0.0.1:42475)
0620 02:34:06.213240 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:06.213250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:06.213712 (+ 462us) server_negotiation.cc:407] Connection header received
0620 02:34:06.213862 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:06.213866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:06.213919 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:06.214011 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:06.214995 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:06.215000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:06.215003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:06.215020 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:06.215062 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:06.215078 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:06.215188 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:06.215327 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 02:34:06.215380 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:34:06.216845 22114 log.cc:824] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:06.219817 22114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf: No bootstrap required, opened a new log
I20260620 02:34:06.228276 22114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c01e5d0744708a0736f1c6f4b5acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 02:34:06.228760 22114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:06.228883 22114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200c01e5d0744708a0736f1c6f4b5acf, State: Initialized, Role: FOLLOWER
I20260620 02:34:06.229676 22114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c01e5d0744708a0736f1c6f4b5acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 02:34:06.229944 22114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:06.230104 22114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:06.230299 22114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:06.232609 22114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c01e5d0744708a0736f1c6f4b5acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 02:34:06.233271 22114 leader_election.cc:302] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200c01e5d0744708a0736f1c6f4b5acf; no voters:
I20260620 02:34:06.233946 22114 leader_election.cc:288] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:06.234107 22124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:06.234827 22124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [term 1 LEADER]: Becoming Leader. State: Replica: 200c01e5d0744708a0736f1c6f4b5acf, State: Running, Role: LEADER
I20260620 02:34:06.235419 22124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c01e5d0744708a0736f1c6f4b5acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 02:34:06.236181 22114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:06.238191 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "200c01e5d0744708a0736f1c6f4b5acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c01e5d0744708a0736f1c6f4b5acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } } }
I20260620 02:34:06.238801 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:06.238276 22126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200c01e5d0744708a0736f1c6f4b5acf. Latest consensus state: current_term: 1 leader_uuid: "200c01e5d0744708a0736f1c6f4b5acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c01e5d0744708a0736f1c6f4b5acf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } } }
I20260620 02:34:06.239596 22126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:06.241338 22132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:06.247586 22132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:06.257968 22132 catalog_manager.cc:1347] Generated new cluster ID: 1537cbb14cac46fba3af1c2dd0db9693
I20260620 02:34:06.258116 22132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:06.272545 22132 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:06.273151 22132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:06.285554 22132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 200c01e5d0744708a0736f1c6f4b5acf: Generated new TSK 0
I20260620 02:34:06.286655 22132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:06.294621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22047
W20260620 02:34:06.296106 22109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:50679
W20260620 02:34:06.296877 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22047: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:34:06.297046 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22047
2026-06-20T02:34:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:06 dist-test-slave-r706 krb5kdc[22145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:06 dist-test-slave-r706 krb5kdc[22145](info): set up 2 sockets
Jun 20 02:34:06 dist-test-slave-r706 krb5kdc[22145](info): commencing operation
krb5kdc: starting...
W20260620 02:34:07.442378 388 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.002s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:07 dist-test-slave-r706 krb5kdc[22145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:07.749307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38629
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:08.115583 22158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:08.115921 22158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:08.116019 22158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:08.123840 22158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:08.123980 22158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:08.124030 22158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:08.124095 22158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:08.124135 22158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:08.134547 22158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:08.135988 22158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:08.137450 22158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:08.150535 22163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:08.150573 22164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:08.150480 22166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:08.151768 22158 server_base.cc:1034] running on GCE node
I20260620 02:34:08.152634 22158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:08.154275 22158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:08.155519 22158 hybrid_clock.cc:648] HybridClock initialized: now 1781922848155469 us; error 66 us; skew 500 ppm
I20260620 02:34:08.155900 22158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:08 dist-test-slave-r706 krb5kdc[22145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:08.163048 22158 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:08.164624 22158 webserver.cc:460] Webserver started at http://127.0.97.62:46881/ using document root <none> and password file <none>
I20260620 02:34:08.165438 22158 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:08.165551 22158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:08.165820 22158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:08.168838 22158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "02a79630252e4df2aee2adbe55e08296"
format_stamp: "Formatted at 2026-06-20 02:34:08 on dist-test-slave-r706"
server_key: "f8ffba1627f9c22e6ccb5652d039230a"
server_key_iv: "4c3b148f3aefc6686ff69215c940717f"
server_key_version: "encryptionkey@0"
I20260620 02:34:08.169667 22158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "02a79630252e4df2aee2adbe55e08296"
format_stamp: "Formatted at 2026-06-20 02:34:08 on dist-test-slave-r706"
server_key: "f8ffba1627f9c22e6ccb5652d039230a"
server_key_iv: "4c3b148f3aefc6686ff69215c940717f"
server_key_version: "encryptionkey@0"
I20260620 02:34:08.175210 22158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:34:08.178952 22173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:08.180423 22158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:34:08.180657 22158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "02a79630252e4df2aee2adbe55e08296"
format_stamp: "Formatted at 2026-06-20 02:34:08 on dist-test-slave-r706"
server_key: "f8ffba1627f9c22e6ccb5652d039230a"
server_key_iv: "4c3b148f3aefc6686ff69215c940717f"
server_key_version: "encryptionkey@0"
I20260620 02:34:08.180866 22158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:08.219710 22158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:08.224220 22158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:08.224557 22158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:08.244771 22158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38629
I20260620 02:34:08.244805 22224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38629 every 8 connection(s)
I20260620 02:34:08.246366 22158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:08.250303 22225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:08.251704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22158
I20260620 02:34:08.251896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:08.252549 388 external_mini_cluster.cc:1428] Setting key d2d5903c0dd3e80446e17c78fa130920
I20260620 02:34:08.265580 22225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296: Bootstrap starting.
Jun 20 02:34:08 dist-test-slave-r706 krb5kdc[22145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:08.269462 22225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:08.270601 22225 log.cc:824] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:08.273732 22225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296: No bootstrap required, opened a new log
I20260620 02:34:08.276955 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:08.254452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55315 (local address 127.0.97.62:38629)
0620 02:34:08.255365 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:08.255377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:08.255421 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:34:08.257274 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:08.257299 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:08.257763 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:08.258132 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:08.259957 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:08.261359 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:08.263284 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:08.264007 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:08.264139 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:08.269061 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:08.269077 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:08.269093 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:08.269136 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:08.272356 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:08.273348 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:08.273355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:08.273362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:08.273523 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:08.274066 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:08.274076 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:08.274078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:08.274643 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:08.274987 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:08.275522 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 02:34:08.275853 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":400,"threads_started":1}
I20260620 02:34:08.281365 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:08.282444 22225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a79630252e4df2aee2adbe55e08296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38629 } }
I20260620 02:34:08.282860 22225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:08.282977 22225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a79630252e4df2aee2adbe55e08296, State: Initialized, Role: FOLLOWER
I20260620 02:34:08.283620 22225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a79630252e4df2aee2adbe55e08296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38629 } }
I20260620 02:34:08.283876 22225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:08.284015 22225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:08.284307 22225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:08.287274 22225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a79630252e4df2aee2adbe55e08296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38629 } }
I20260620 02:34:08.287855 22225 leader_election.cc:302] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a79630252e4df2aee2adbe55e08296; no voters:
I20260620 02:34:08.288461 22225 leader_election.cc:288] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:08.288631 22235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:08.292644 22225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:08.293489 22235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [term 1 LEADER]: Becoming Leader. State: Replica: 02a79630252e4df2aee2adbe55e08296, State: Running, Role: LEADER
I20260620 02:34:08.294217 22235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a79630252e4df2aee2adbe55e08296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38629 } }
I20260620 02:34:08.300501 22236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a79630252e4df2aee2adbe55e08296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a79630252e4df2aee2adbe55e08296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38629 } } }
I20260620 02:34:08.301597 22236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:08.301477 22237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a79630252e4df2aee2adbe55e08296. Latest consensus state: current_term: 1 leader_uuid: "02a79630252e4df2aee2adbe55e08296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a79630252e4df2aee2adbe55e08296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38629 } } }
I20260620 02:34:08.301833 22237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296 [sys.catalog]: This master's current role is: LEADER
W20260620 02:34:08.308367 22249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:34:08.308533 22249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:34:08.308720 22250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:08.308813 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:08.286131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44289 (local address 127.0.97.62:38629)
0620 02:34:08.286327 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:08.286334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:08.286818 (+ 484us) server_negotiation.cc:407] Connection header received
0620 02:34:08.286971 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:08.286979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:08.287070 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:08.287204 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:08.289143 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:08.290795 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:08.293320 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:08.294115 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:08.294296 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:08.298089 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:08.298098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:08.298106 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:08.298166 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:08.304187 (+ 6021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:08.305472 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:08.305476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:08.305478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:08.305607 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:08.306227 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:08.306232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:08.306234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:08.306677 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:08.307957 (+ 1280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:08.308163 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 02:34:08.308362 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:34:08.310672 22250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:08.319243 22250 catalog_manager.cc:1347] Generated new cluster ID: 15139cd1e3944bf39519b81c50ae8310
I20260620 02:34:08.319384 22250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:08.334996 22250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:08.336413 22250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:08.348268 22250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02a79630252e4df2aee2adbe55e08296: Generated new TSK 0
I20260620 02:34:08.349311 22250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:08.357470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22158
I20260620 02:34:08.358806 22220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:08.453183 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22158
W20260620 02:34:08.471900 22143 connection.cc:535] client connection to 127.0.97.62:38629 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38629): Connection reset by peer (error 104)
2026-06-20T02:34:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2155 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:08 dist-test-slave-r706 krb5kdc[22257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:08 dist-test-slave-r706 krb5kdc[22257](info): set up 2 sockets
Jun 20 02:34:08 dist-test-slave-r706 krb5kdc[22257](info): commencing operation
krb5kdc: starting...
W20260620 02:34:09.589710 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:09 dist-test-slave-r706 krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:09.902823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38327
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:10.250953 22270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:10.251276 22270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:10.251443 22270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:10.258841 22270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:10.258977 22270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:10.259080 22270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:10.259203 22270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:10.259298 22270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:10.269671 22270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:10.271086 22270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:10.272424 22270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:10.285207 22278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:10.285269 22275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:10.285292 22276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:10.286253 22270 server_base.cc:1034] running on GCE node
I20260620 02:34:10.287140 22270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:10.288681 22270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:10.289888 22270 hybrid_clock.cc:648] HybridClock initialized: now 1781922850289812 us; error 77 us; skew 500 ppm
I20260620 02:34:10.290261 22270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:10 dist-test-slave-r706 krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:10.293640 22270 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:10.295243 22270 webserver.cc:460] Webserver started at http://127.0.97.62:46621/ using document root <none> and password file <none>
I20260620 02:34:10.295977 22270 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:10.296139 22270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:10.296446 22270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:10.299495 22270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "95bfb8f6fd6a4e11ab818014771d09bc"
format_stamp: "Formatted at 2026-06-20 02:34:10 on dist-test-slave-r706"
server_key: "01902bb1964312376dbcfa82853a70da"
server_key_iv: "746a3451833a48667c407bafc6da0404"
server_key_version: "encryptionkey@0"
I20260620 02:34:10.300349 22270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "95bfb8f6fd6a4e11ab818014771d09bc"
format_stamp: "Formatted at 2026-06-20 02:34:10 on dist-test-slave-r706"
server_key: "01902bb1964312376dbcfa82853a70da"
server_key_iv: "746a3451833a48667c407bafc6da0404"
server_key_version: "encryptionkey@0"
I20260620 02:34:10.305887 22270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 02:34:10.309681 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:10.311117 22270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:34:10.311308 22270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "95bfb8f6fd6a4e11ab818014771d09bc"
format_stamp: "Formatted at 2026-06-20 02:34:10 on dist-test-slave-r706"
server_key: "01902bb1964312376dbcfa82853a70da"
server_key_iv: "746a3451833a48667c407bafc6da0404"
server_key_version: "encryptionkey@0"
I20260620 02:34:10.311525 22270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:10.336910 22270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:10.341367 22270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:10.341692 22270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:10.360610 22270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38327
I20260620 02:34:10.360643 22336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38327 every 8 connection(s)
I20260620 02:34:10.362089 22270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:10.362886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22270
I20260620 02:34:10.363102 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:10.363629 388 external_mini_cluster.cc:1428] Setting key 2bba019bbc69381d4796d0a8af105af0
I20260620 02:34:10.366978 22337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:10.379674 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:10.365556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33377 (local address 127.0.97.62:38327)
0620 02:34:10.366392 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:10.366408 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:10.366460 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:34:10.368858 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:10.368885 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:10.369413 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:10.369848 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:10.370396 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:10.372468 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:10.374056 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:10.375120 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:10.375310 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:10.376651 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:10.376662 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:10.376684 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:10.377498 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:10.377602 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:10.377634 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:10.377808 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:10.378096 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 02:34:10.378294 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":355,"threads_started":1}
I20260620 02:34:10.381618 22337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc: Bootstrap starting.
I20260620 02:34:10.384184 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:10.385679 22337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:10.386726 22337 log.cc:824] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:10.390208 22337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc: No bootstrap required, opened a new log
I20260620 02:34:10.395809 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:10.388855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45561 (local address 127.0.97.62:38327)
0620 02:34:10.389072 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:10.389080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:10.389455 (+ 375us) server_negotiation.cc:407] Connection header received
0620 02:34:10.389614 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:10.389622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:10.389704 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:10.389828 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:10.390326 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:10.391559 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:10.392935 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:10.393699 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:10.393845 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:10.394972 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:10.394980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:10.394985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:10.395010 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:10.395070 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:10.395090 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:10.395224 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:10.395379 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 02:34:10.395453 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:34:10.399509 22337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260620 02:34:10.400004 22337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:10.400122 22337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bfb8f6fd6a4e11ab818014771d09bc, State: Initialized, Role: FOLLOWER
I20260620 02:34:10.400712 22337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260620 02:34:10.400974 22337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:10.401208 22337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:10.401408 22337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:10.403687 22337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260620 02:34:10.404237 22337 leader_election.cc:302] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95bfb8f6fd6a4e11ab818014771d09bc; no voters:
I20260620 02:34:10.404973 22337 leader_election.cc:288] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:10.405129 22347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:10.405983 22347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [term 1 LEADER]: Becoming Leader. State: Replica: 95bfb8f6fd6a4e11ab818014771d09bc, State: Running, Role: LEADER
I20260620 02:34:10.406543 22347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260620 02:34:10.407308 22337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:10.409495 22348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } } }
I20260620 02:34:10.409909 22348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:10.412093 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bfb8f6fd6a4e11ab818014771d09bc. Latest consensus state: current_term: 1 leader_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfb8f6fd6a4e11ab818014771d09bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } } }
I20260620 02:34:10.412427 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:10.413875 22353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:10.419461 22353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:10.429270 22353 catalog_manager.cc:1347] Generated new cluster ID: a6c6567c1e1f40f2b79d25b6ff5538b8
I20260620 02:34:10.429412 22353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:10.440670 22353 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:10.442076 22353 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:10.462416 22353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95bfb8f6fd6a4e11ab818014771d09bc: Generated new TSK 0
I20260620 02:34:10.463358 22353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:10.475193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22270
W20260620 02:34:10.476292 22332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33377
W20260620 02:34:10.477155 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:34:10.477301 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22270
W20260620 02:34:10.496111 22255 connection.cc:535] client connection to 127.0.97.62:38327 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38327): Connection reset by peer (error 104)
2026-06-20T02:34:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:10 dist-test-slave-r706 krb5kdc[22368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:10 dist-test-slave-r706 krb5kdc[22368](info): set up 2 sockets
Jun 20 02:34:10 dist-test-slave-r706 krb5kdc[22368](info): commencing operation
krb5kdc: starting...
W20260620 02:34:11.613677 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:11 dist-test-slave-r706 krb5kdc[22368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:11.912218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43445
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:12.260217 22381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:12.260545 22381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:12.260725 22381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:12.268385 22381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:12.268510 22381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:12.268620 22381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:12.268725 22381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:12.268777 22381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:12.278861 22381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:12.280238 22381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:12.281599 22381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:12.293895 22386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:12.293902 22387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:12.294554 22389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:12.294646 22381 server_base.cc:1034] running on GCE node
I20260620 02:34:12.295576 22381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:12.297214 22381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:12.298455 22381 hybrid_clock.cc:648] HybridClock initialized: now 1781922852298399 us; error 57 us; skew 500 ppm
I20260620 02:34:12.298847 22381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:12 dist-test-slave-r706 krb5kdc[22368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:12.302181 22381 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:12.303810 22381 webserver.cc:460] Webserver started at http://127.0.97.62:45363/ using document root <none> and password file <none>
I20260620 02:34:12.304533 22381 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:12.304698 22381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:12.305085 22381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:12.308205 22381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "99eb149c2f2d4695913602f2ffed8bcb"
format_stamp: "Formatted at 2026-06-20 02:34:12 on dist-test-slave-r706"
server_key: "6f0d5a67066244edd2fab0a23f61b904"
server_key_iv: "131447fa3b181838fbae33b0fd5f41b2"
server_key_version: "encryptionkey@0"
I20260620 02:34:12.309054 22381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "99eb149c2f2d4695913602f2ffed8bcb"
format_stamp: "Formatted at 2026-06-20 02:34:12 on dist-test-slave-r706"
server_key: "6f0d5a67066244edd2fab0a23f61b904"
server_key_iv: "131447fa3b181838fbae33b0fd5f41b2"
server_key_version: "encryptionkey@0"
I20260620 02:34:12.314366 22381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:34:12.317876 22396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:12.319242 22381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:34:12.319438 22381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "99eb149c2f2d4695913602f2ffed8bcb"
format_stamp: "Formatted at 2026-06-20 02:34:12 on dist-test-slave-r706"
server_key: "6f0d5a67066244edd2fab0a23f61b904"
server_key_iv: "131447fa3b181838fbae33b0fd5f41b2"
server_key_version: "encryptionkey@0"
I20260620 02:34:12.319666 22381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:12.325675 22381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:12.329734 22381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:12.330051 22381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:12.349436 22381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43445
I20260620 02:34:12.349480 22447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43445 every 8 connection(s)
I20260620 02:34:12.350986 22381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:12.353130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22381
I20260620 02:34:12.353310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:12.353861 388 external_mini_cluster.cc:1428] Setting key 4527704d2c486ec7f8d09a88154b932e
I20260620 02:34:12.355914 22448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:12.362260 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:12.355809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47293 (local address 127.0.97.62:43445)
0620 02:34:12.356705 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:12.356717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:12.356759 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:34:12.358504 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:12.358527 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:12.358890 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:12.359327 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:12.360423 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:12.360429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:12.360441 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:12.360455 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:12.360665 (+ 210us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:12.360688 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:12.360943 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:12.361124 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 02:34:12.361312 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":404,"threads_started":1}
I20260620 02:34:12.366222 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:12.368948 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb: Bootstrap starting.
I20260620 02:34:12.373279 22448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:12.374004 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:12.370847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54211 (local address 127.0.97.62:43445)
0620 02:34:12.371078 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:12.371084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:12.371460 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:34:12.371641 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:12.371645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:12.371694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:12.371778 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:12.373084 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:12.373090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:12.373177 (+ 87us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:12.373208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:12.373266 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:12.373297 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:12.373466 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:12.373655 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 02:34:12.373722 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:34:12.374631 22448 log.cc:824] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:12.378276 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb: No bootstrap required, opened a new log
I20260620 02:34:12.387444 22448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99eb149c2f2d4695913602f2ffed8bcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43445 } }
I20260620 02:34:12.387851 22448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:12.387938 22448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99eb149c2f2d4695913602f2ffed8bcb, State: Initialized, Role: FOLLOWER
I20260620 02:34:12.388465 22448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99eb149c2f2d4695913602f2ffed8bcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43445 } }
I20260620 02:34:12.388662 22448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:12.388762 22448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:12.388912 22448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:12.392562 22448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99eb149c2f2d4695913602f2ffed8bcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43445 } }
I20260620 02:34:12.393158 22448 leader_election.cc:302] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99eb149c2f2d4695913602f2ffed8bcb; no voters:
I20260620 02:34:12.393813 22448 leader_election.cc:288] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:12.393989 22458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:12.394672 22458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [term 1 LEADER]: Becoming Leader. State: Replica: 99eb149c2f2d4695913602f2ffed8bcb, State: Running, Role: LEADER
I20260620 02:34:12.395265 22458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99eb149c2f2d4695913602f2ffed8bcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43445 } }
I20260620 02:34:12.396075 22448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:12.398114 22459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99eb149c2f2d4695913602f2ffed8bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99eb149c2f2d4695913602f2ffed8bcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43445 } } }
I20260620 02:34:12.400223 22459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:12.400393 22460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99eb149c2f2d4695913602f2ffed8bcb. Latest consensus state: current_term: 1 leader_uuid: "99eb149c2f2d4695913602f2ffed8bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99eb149c2f2d4695913602f2ffed8bcb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43445 } } }
I20260620 02:34:12.401933 22460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:12.402199 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:12.408749 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:12.419251 22468 catalog_manager.cc:1347] Generated new cluster ID: addc3afdd63c4b42a5e5a5918f767bf0
I20260620 02:34:12.419404 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:12.445365 22468 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:12.445832 22468 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:12.453778 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99eb149c2f2d4695913602f2ffed8bcb: Generated new TSK 0
I20260620 02:34:12.454721 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:12.473289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22381
W20260620 02:34:12.474335 22443 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47293
W20260620 02:34:12.475075 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22381: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:34:12.475229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22381
2026-06-20T02:34:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:12 dist-test-slave-r706 krb5kdc[22479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:12 dist-test-slave-r706 krb5kdc[22479](info): set up 2 sockets
Jun 20 02:34:12 dist-test-slave-r706 krb5kdc[22479](info): commencing operation
krb5kdc: starting...
W20260620 02:34:13.665576 388 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.005s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:13 dist-test-slave-r706 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260620 02:34:14.010176 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.101s user 0.000s sys 0.033s
I20260620 02:34:14.011482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37667
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:14.370738 22492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:14.371048 22492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:14.371229 22492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:14.378742 22492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:14.378862 22492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:14.378965 22492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:14.379115 22492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:14.379192 22492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:14.389261 22492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:14.390676 22492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:14.392218 22492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:14.405503 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:14.405527 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:14.405453 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:14.406471 22492 server_base.cc:1034] running on GCE node
I20260620 02:34:14.407336 22492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:14.408864 22492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:14.410148 22492 hybrid_clock.cc:648] HybridClock initialized: now 1781922854410113 us; error 48 us; skew 500 ppm
I20260620 02:34:14.410566 22492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:14 dist-test-slave-r706 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:14.418924 22492 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:14.420643 22492 webserver.cc:460] Webserver started at http://127.0.97.62:43583/ using document root <none> and password file <none>
I20260620 02:34:14.421432 22492 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:14.421550 22492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:14.421818 22492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:14.425089 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "12e81bea416b43ec9040a83d8f7ee4d2"
format_stamp: "Formatted at 2026-06-20 02:34:14 on dist-test-slave-r706"
server_key: "4cb4c126d89e2fbf0f67a5b713a943ce"
server_key_iv: "401c1c5ba4bb6e2e8cd54e0f74f8a3b0"
server_key_version: "encryptionkey@0"
I20260620 02:34:14.425936 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "12e81bea416b43ec9040a83d8f7ee4d2"
format_stamp: "Formatted at 2026-06-20 02:34:14 on dist-test-slave-r706"
server_key: "4cb4c126d89e2fbf0f67a5b713a943ce"
server_key_iv: "401c1c5ba4bb6e2e8cd54e0f74f8a3b0"
server_key_version: "encryptionkey@0"
I20260620 02:34:14.431447 22492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:34:14.435251 22507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:14.436712 22492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:34:14.436930 22492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "12e81bea416b43ec9040a83d8f7ee4d2"
format_stamp: "Formatted at 2026-06-20 02:34:14 on dist-test-slave-r706"
server_key: "4cb4c126d89e2fbf0f67a5b713a943ce"
server_key_iv: "401c1c5ba4bb6e2e8cd54e0f74f8a3b0"
server_key_version: "encryptionkey@0"
I20260620 02:34:14.437188 22492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:14.503873 22492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:14.508351 22492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:14.508693 22492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:14.528759 22492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37667
I20260620 02:34:14.528813 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37667 every 8 connection(s)
I20260620 02:34:14.530269 22492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:14.531785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22492
I20260620 02:34:14.531978 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:14.532521 388 external_mini_cluster.cc:1428] Setting key 669eeb0cf2b40595254d8f9d398369e4
I20260620 02:34:14.535246 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:34:14 dist-test-slave-r706 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:14.549892 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2: Bootstrap starting.
I20260620 02:34:14.553352 22559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:14.554677 22559 log.cc:824] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:14.558230 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2: No bootstrap required, opened a new log
I20260620 02:34:14.558686 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:14.534534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60969 (local address 127.0.97.62:37667)
0620 02:34:14.535432 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:14.535444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:14.535487 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:34:14.537464 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:14.537487 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:14.537911 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:14.538279 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:14.540166 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:14.541873 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:14.543741 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:14.544454 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:14.544625 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:14.549942 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:14.549952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:14.550005 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:14.550091 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:14.553857 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:14.554783 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:14.554790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:14.554797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:14.555026 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:14.555530 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:14.555534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:14.555537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:14.556284 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:14.556515 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:14.557136 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 02:34:14.557591 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":390,"threads_started":1}
I20260620 02:34:14.562927 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:14.568789 22559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37667 } }
I20260620 02:34:14.569360 22559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:14.569485 22559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e81bea416b43ec9040a83d8f7ee4d2, State: Initialized, Role: FOLLOWER
I20260620 02:34:14.570149 22559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37667 } }
I20260620 02:34:14.570396 22559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:14.570542 22559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:14.570732 22559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:14.573871 22559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37667 } }
I20260620 02:34:14.574479 22559 leader_election.cc:302] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e81bea416b43ec9040a83d8f7ee4d2; no voters:
I20260620 02:34:14.575212 22559 leader_election.cc:288] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:14.575328 22569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:14.577306 22569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [term 1 LEADER]: Becoming Leader. State: Replica: 12e81bea416b43ec9040a83d8f7ee4d2, State: Running, Role: LEADER
I20260620 02:34:14.577993 22559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:14.578032 22569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37667 } }
I20260620 02:34:14.581472 22571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e81bea416b43ec9040a83d8f7ee4d2. Latest consensus state: current_term: 1 leader_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37667 } } }
I20260620 02:34:14.581959 22571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:14.587098 22570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e81bea416b43ec9040a83d8f7ee4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37667 } } }
I20260620 02:34:14.587463 22570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:14.587953 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:14.568350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44843 (local address 127.0.97.62:37667)
0620 02:34:14.568569 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:14.568575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:14.569182 (+ 607us) server_negotiation.cc:407] Connection header received
0620 02:34:14.569359 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:14.569364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:14.569466 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:14.569624 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:14.571367 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:14.572542 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:14.574149 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:14.574626 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:14.574776 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:14.578059 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:14.578067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:14.578072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:14.578113 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:14.582683 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:14.583755 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:14.583760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:14.583763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:14.583922 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:14.584543 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:14.584551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:14.584557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:14.585062 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:14.586694 (+ 1632us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:14.586925 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 02:34:14.587270 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:34:14.589664 22576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:14.594363 22576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:14.604113 22576 catalog_manager.cc:1347] Generated new cluster ID: 95e9754a90b84e1f91b22678d5f061c1
I20260620 02:34:14.604261 22576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:14.617643 22576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:14.619076 22576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:14.630570 22576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12e81bea416b43ec9040a83d8f7ee4d2: Generated new TSK 0
I20260620 02:34:14.631538 22576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:14.646474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22492
I20260620 02:34:14.647984 22554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:14.742516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22492
2026-06-20T02:34:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:14 dist-test-slave-r706 krb5kdc[22591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:14 dist-test-slave-r706 krb5kdc[22591](info): set up 2 sockets
Jun 20 02:34:14 dist-test-slave-r706 krb5kdc[22591](info): commencing operation
krb5kdc: starting...
W20260620 02:34:15.969758 388 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.005s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:16 dist-test-slave-r706 krb5kdc[22591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:16.284950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36945
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:16.645762 22604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:16.646188 22604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:16.646379 22604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:16.654422 22604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:16.654603 22604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:16.654734 22604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:16.654867 22604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:16.654960 22604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:16.665951 22604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:16.667455 22604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:16.668901 22604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:16.681964 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:16.682018 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:16.682049 22612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:16.682870 22604 server_base.cc:1034] running on GCE node
I20260620 02:34:16.683808 22604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:16.685583 22604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:16.686889 22604 hybrid_clock.cc:648] HybridClock initialized: now 1781922856686832 us; error 54 us; skew 500 ppm
I20260620 02:34:16.687307 22604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:16 dist-test-slave-r706 krb5kdc[22591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:16.690804 22604 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:16.692528 22604 webserver.cc:460] Webserver started at http://127.0.97.62:37805/ using document root <none> and password file <none>
I20260620 02:34:16.693356 22604 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:16.693522 22604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:16.693851 22604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:16.697153 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "f0df96126d484b62bba474194579f4ce"
format_stamp: "Formatted at 2026-06-20 02:34:16 on dist-test-slave-r706"
server_key: "135c129dffcfae2b7b671c51c56f7797"
server_key_iv: "5cca603b9c792b719631512100f13815"
server_key_version: "encryptionkey@0"
I20260620 02:34:16.697989 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0df96126d484b62bba474194579f4ce"
format_stamp: "Formatted at 2026-06-20 02:34:16 on dist-test-slave-r706"
server_key: "135c129dffcfae2b7b671c51c56f7797"
server_key_iv: "5cca603b9c792b719631512100f13815"
server_key_version: "encryptionkey@0"
I20260620 02:34:16.704272 22604 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:34:16.708568 22619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:16.710196 22604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:34:16.710433 22604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "f0df96126d484b62bba474194579f4ce"
format_stamp: "Formatted at 2026-06-20 02:34:16 on dist-test-slave-r706"
server_key: "135c129dffcfae2b7b671c51c56f7797"
server_key_iv: "5cca603b9c792b719631512100f13815"
server_key_version: "encryptionkey@0"
I20260620 02:34:16.710681 22604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:16.741178 22604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:16.746407 22604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:16.746740 22604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:16.769434 22604 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36945
I20260620 02:34:16.769503 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36945 every 8 connection(s)
I20260620 02:34:16.770975 22604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:16.775166 22671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:16.776989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22604
I20260620 02:34:16.777231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:16.777812 388 external_mini_cluster.cc:1428] Setting key 397638b7d5e58401514d367bef455dbd
I20260620 02:34:16.790833 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce: Bootstrap starting.
I20260620 02:34:16.792501 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:16.780008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42717 (local address 127.0.97.62:36945)
0620 02:34:16.781061 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:16.781083 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:16.781133 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:34:16.782788 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:16.782817 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:16.783329 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:16.783697 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:16.784313 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:16.785733 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:16.787724 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:16.788470 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:16.788609 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:16.789980 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:16.789987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:16.790002 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:16.790568 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:16.790661 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:16.790686 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:16.790778 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:16.791181 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 02:34:16.791480 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":515,"threads_started":1}
I20260620 02:34:16.794358 22671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:16.795640 22671 log.cc:824] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:16.796363 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:16.798609 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce: No bootstrap required, opened a new log
I20260620 02:34:16.808393 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:16.800989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33949 (local address 127.0.97.62:36945)
0620 02:34:16.801233 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:16.801240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:16.801595 (+ 355us) server_negotiation.cc:407] Connection header received
0620 02:34:16.801782 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:16.801786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:16.801854 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:16.801976 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:16.802741 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:16.803572 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:16.805627 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:16.806116 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:16.806233 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:16.807538 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:16.807543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:16.807546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:16.807575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:16.807620 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:16.807637 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:16.807722 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:16.807932 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 02:34:16.808011 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:34:16.808893 22671 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df96126d484b62bba474194579f4ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260620 02:34:16.809533 22671 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:16.809651 22671 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0df96126d484b62bba474194579f4ce, State: Initialized, Role: FOLLOWER
I20260620 02:34:16.810390 22671 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df96126d484b62bba474194579f4ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260620 02:34:16.810621 22671 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:16.810729 22671 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:16.810856 22671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:16.813189 22671 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df96126d484b62bba474194579f4ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260620 02:34:16.813768 22671 leader_election.cc:302] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0df96126d484b62bba474194579f4ce; no voters:
I20260620 02:34:16.814622 22671 leader_election.cc:288] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:16.814744 22681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:16.816190 22681 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [term 1 LEADER]: Becoming Leader. State: Replica: f0df96126d484b62bba474194579f4ce, State: Running, Role: LEADER
I20260620 02:34:16.816761 22681 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df96126d484b62bba474194579f4ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } }
I20260620 02:34:16.817065 22671 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:16.819566 22683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0df96126d484b62bba474194579f4ce. Latest consensus state: current_term: 1 leader_uuid: "f0df96126d484b62bba474194579f4ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df96126d484b62bba474194579f4ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } } }
I20260620 02:34:16.819965 22683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:16.820518 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0df96126d484b62bba474194579f4ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0df96126d484b62bba474194579f4ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36945 } } }
I20260620 02:34:16.820840 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:16.828620 22688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:16.834821 22688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:16.844472 22688 catalog_manager.cc:1347] Generated new cluster ID: f7c8df5d5e6f4504bee4caf4251a12d5
I20260620 02:34:16.844623 22688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:16.857484 22688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:16.858911 22688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:16.868925 22688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0df96126d484b62bba474194579f4ce: Generated new TSK 0
I20260620 02:34:16.870036 22688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:16.875777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22604
W20260620 02:34:16.877070 22666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42717
W20260620 02:34:16.877965 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22604: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:34:16.878168 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22604
2026-06-20T02:34:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2135 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:16 dist-test-slave-r706 krb5kdc[22703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:16 dist-test-slave-r706 krb5kdc[22703](info): set up 2 sockets
Jun 20 02:34:16 dist-test-slave-r706 krb5kdc[22703](info): commencing operation
krb5kdc: starting...
W20260620 02:34:18.009713 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:18 dist-test-slave-r706 krb5kdc[22703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:18.341485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33903
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:18.698609 22716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:18.698956 22716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:18.699123 22716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:18.707052 22716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:18.707219 22716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:18.707340 22716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:18.707468 22716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:18.707547 22716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:18.718331 22716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:18.719777 22716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:18.721287 22716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:18.734732 22724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:18.734732 22722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:18.734834 22721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:18.735731 22716 server_base.cc:1034] running on GCE node
I20260620 02:34:18.736582 22716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:18.738262 22716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:18.739539 22716 hybrid_clock.cc:648] HybridClock initialized: now 1781922858739446 us; error 69 us; skew 500 ppm
I20260620 02:34:18.739933 22716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:18 dist-test-slave-r706 krb5kdc[22703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:18.743709 22716 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:18.745558 22716 webserver.cc:460] Webserver started at http://127.0.97.62:42277/ using document root <none> and password file <none>
I20260620 02:34:18.746384 22716 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:18.746548 22716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:18.746881 22716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:18.750259 22716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "08feb5e669564e6a9bf7f432656d545b"
format_stamp: "Formatted at 2026-06-20 02:34:18 on dist-test-slave-r706"
server_key: "82543a9e4b39fd78e8c83076a5cc46b3"
server_key_iv: "624a0b3e6dedf55eb8f2d0286d76d63d"
server_key_version: "encryptionkey@0"
I20260620 02:34:18.751145 22716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "08feb5e669564e6a9bf7f432656d545b"
format_stamp: "Formatted at 2026-06-20 02:34:18 on dist-test-slave-r706"
server_key: "82543a9e4b39fd78e8c83076a5cc46b3"
server_key_iv: "624a0b3e6dedf55eb8f2d0286d76d63d"
server_key_version: "encryptionkey@0"
I20260620 02:34:18.757161 22716 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:34:18.761564 22731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:18.763160 22716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:34:18.763391 22716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "08feb5e669564e6a9bf7f432656d545b"
format_stamp: "Formatted at 2026-06-20 02:34:18 on dist-test-slave-r706"
server_key: "82543a9e4b39fd78e8c83076a5cc46b3"
server_key_iv: "624a0b3e6dedf55eb8f2d0286d76d63d"
server_key_version: "encryptionkey@0"
I20260620 02:34:18.763652 22716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:18.770035 22716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:18.774852 22716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:18.775204 22716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:18.797317 22716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33903
I20260620 02:34:18.797379 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33903 every 8 connection(s)
I20260620 02:34:18.798966 22716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:18.802713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22716
I20260620 02:34:18.802927 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:18.803366 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:18.803505 388 external_mini_cluster.cc:1428] Setting key a87e10b46113d752c2e21a5c8fe66c99
I20260620 02:34:18.811524 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:18.805356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45795 (local address 127.0.97.62:33903)
0620 02:34:18.806117 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:18.806130 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:18.806171 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:34:18.807816 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:18.807839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:18.808211 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:18.808662 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:18.809825 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:18.809832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:18.809844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:18.809857 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:18.809937 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:18.809967 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:18.810229 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:18.810383 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:34:18.810584 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":302,"threads_started":1}
I20260620 02:34:18.815625 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:18.817886 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b: Bootstrap starting.
I20260620 02:34:18.821761 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:18.823338 22783 log.cc:824] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:18.823967 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:18.820963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53803 (local address 127.0.97.62:33903)
0620 02:34:18.821198 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:18.821203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:18.821729 (+ 526us) server_negotiation.cc:407] Connection header received
0620 02:34:18.821983 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:18.821989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:18.822051 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:18.822191 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:18.823272 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:18.823276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:18.823280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:34:18.823297 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:18.823346 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:34:18.823362 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:18.823508 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:18.823592 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 02:34:18.823660 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 02:34:18.826651 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b: No bootstrap required, opened a new log
I20260620 02:34:18.834875 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08feb5e669564e6a9bf7f432656d545b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33903 } }
I20260620 02:34:18.835342 22783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:34:18.835451 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08feb5e669564e6a9bf7f432656d545b, State: Initialized, Role: FOLLOWER
I20260620 02:34:18.836036 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08feb5e669564e6a9bf7f432656d545b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33903 } }
I20260620 02:34:18.836256 22783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:34:18.836369 22783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:34:18.836563 22783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:18.839286 22783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08feb5e669564e6a9bf7f432656d545b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33903 } }
I20260620 02:34:18.839800 22783 leader_election.cc:302] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08feb5e669564e6a9bf7f432656d545b; no voters:
I20260620 02:34:18.840433 22783 leader_election.cc:288] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:34:18.840623 22793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:18.842350 22793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [term 1 LEADER]: Becoming Leader. State: Replica: 08feb5e669564e6a9bf7f432656d545b, State: Running, Role: LEADER
I20260620 02:34:18.842883 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:18.843037 22793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08feb5e669564e6a9bf7f432656d545b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33903 } }
I20260620 02:34:18.846529 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08feb5e669564e6a9bf7f432656d545b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08feb5e669564e6a9bf7f432656d545b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33903 } } }
I20260620 02:34:18.846570 22795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08feb5e669564e6a9bf7f432656d545b. Latest consensus state: current_term: 1 leader_uuid: "08feb5e669564e6a9bf7f432656d545b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08feb5e669564e6a9bf7f432656d545b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33903 } } }
I20260620 02:34:18.847043 22795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:18.847016 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:18.850258 22801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:18.857463 22801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:18.879884 22801 catalog_manager.cc:1347] Generated new cluster ID: db60fd2dea734623847e58881d9dd18e
I20260620 02:34:18.880039 22801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:18.918978 22801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:18.919497 22801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:18.929931 22801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08feb5e669564e6a9bf7f432656d545b: Generated new TSK 0
I20260620 02:34:18.930932 22801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:18.943542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22716
W20260620 02:34:18.945679 22778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45795
W20260620 02:34:18.946508 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22716: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:34:18.946693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22716
2026-06-20T02:34:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2074 ms)
[----------] 18 tests from AuthTokenIssuingTest (37755 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:19 dist-test-slave-r706 krb5kdc[22814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:19 dist-test-slave-r706 krb5kdc[22814](info): set up 2 sockets
Jun 20 02:34:19 dist-test-slave-r706 krb5kdc[22814](info): commencing operation
krb5kdc: starting...
W20260620 02:34:20.105930 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:20 dist-test-slave-r706 krb5kdc[22814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:20.425100 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33825
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33825,127.0.97.61:39671,127.0.97.60:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:20.781093 22827 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:34:20.781467 22827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:20.781543 22827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:20.781630 22827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:20.789631 22827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:20.789757 22827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:20.789827 22827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:20.789880 22827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:20.789945 22827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:20.800720 22827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33825,127.0.97.61:39671,127.0.97.60:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:20.802148 22827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:20.803476 22827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:20.816242 22832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:20.816099 22833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:20.816074 22835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:20.816598 22827 server_base.cc:1034] running on GCE node
I20260620 02:34:20.817713 22827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:20.819250 22827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:20.820555 22827 hybrid_clock.cc:648] HybridClock initialized: now 1781922860820474 us; error 46 us; skew 500 ppm
I20260620 02:34:20.820962 22827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:20 dist-test-slave-r706 krb5kdc[22814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:20.828727 22827 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:20.830526 22827 webserver.cc:460] Webserver started at http://127.0.97.62:34471/ using document root <none> and password file <none>
I20260620 02:34:20.831264 22827 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:20.831362 22827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:20.831668 22827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:20.835062 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "9793cc81eb254b008e6a44278b8dc367"
format_stamp: "Formatted at 2026-06-20 02:34:20 on dist-test-slave-r706"
server_key: "83289c6c8b337d9f88de4425452b68b6"
server_key_iv: "c70e8beb165d801fa42b6ec456666366"
server_key_version: "encryptionkey@0"
I20260620 02:34:20.835870 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "9793cc81eb254b008e6a44278b8dc367"
format_stamp: "Formatted at 2026-06-20 02:34:20 on dist-test-slave-r706"
server_key: "83289c6c8b337d9f88de4425452b68b6"
server_key_iv: "c70e8beb165d801fa42b6ec456666366"
server_key_version: "encryptionkey@0"
I20260620 02:34:20.841591 22827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:34:20.845427 22842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:20.846871 22827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:34:20.847097 22827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "9793cc81eb254b008e6a44278b8dc367"
format_stamp: "Formatted at 2026-06-20 02:34:20 on dist-test-slave-r706"
server_key: "83289c6c8b337d9f88de4425452b68b6"
server_key_iv: "c70e8beb165d801fa42b6ec456666366"
server_key_version: "encryptionkey@0"
I20260620 02:34:20.847342 22827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:20.886567 22827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:20.891276 22827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:20.891625 22827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:20.912388 22827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33825
I20260620 02:34:20.912407 22893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33825 every 8 connection(s)
I20260620 02:34:20.913938 22827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:20.917251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22827
I20260620 02:34:20.917490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:20.917855 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:20.918200 388 external_mini_cluster.cc:1428] Setting key a902b646a11957b5a2f46e0f6f01429c
I20260620 02:34:20.924582 22894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:20 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 02:34:20.949656 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:20.928034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33825 (local address 127.0.0.1:41862)
0620 02:34:20.929545 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:20.929587 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:20.931396 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:20.932344 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:20.932350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:20.933208 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:20.933568 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:20.933586 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:20.935385 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:20.935391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:20.936577 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:20.936590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:20.937670 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:20.937677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:20.937857 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:20.938624 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:20.938660 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:20.941336 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:20.944792 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:20.944806 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:20.944820 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:20.945322 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:20.945765 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:20.945769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:20.945773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:20.945940 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:20.946938 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:20.946946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:20.947209 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 02:34:20.947545 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 02:34:20.948218 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"mutex_wait_us":185,"thread_start_us":1064,"threads_started":1}
I20260620 02:34:20.949676 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:20.928034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.0.97.62:33825)
0620 02:34:20.929183 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:20.929202 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:20.929724 (+ 522us) server_negotiation.cc:407] Connection header received
0620 02:34:20.931851 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:20.931857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:20.932106 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:20.932336 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:20.933799 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:20.935157 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:20.936738 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:20.937421 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:20.937644 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:20.941592 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:20.941601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:20.941616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:20.941647 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:20.944634 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:20.945468 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:20.945482 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:20.945485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:20.945633 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:20.946062 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:20.946067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:20.946070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:20.946713 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:20.947026 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:20.947693 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 02:34:20.948092 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"spinlock_wait_cycles":37760,"thread_start_us":611,"threads_started":1}
I20260620 02:34:20.953141 22894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:20.954569 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:20.953976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34866)
0620 02:34:20.954248 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:20.954398 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 02:34:20.954981 22844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
I20260620 02:34:20.957047 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:20.956503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34876)
0620 02:34:20.956728 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:20.956826 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 02:34:20.957267 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39671: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:34:20.986503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39671
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33825,127.0.97.61:39671,127.0.97.60:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:34:21.006783 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } attempt: 1
I20260620 02:34:21.008126 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.007640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34890)
0620 02:34:21.007857 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.007959 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:34:21.009260 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.008848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34904)
0620 02:34:21.009055 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.009124 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:34:21.009675 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39671: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
I20260620 02:34:21.078058 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } attempt: 2
I20260620 02:34:21.079336 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.078830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34920)
0620 02:34:21.079109 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.079194 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 02:34:21.080380 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.080026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34934)
0620 02:34:21.080168 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.080255 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 02:34:21.080683 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39671: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
I20260620 02:34:21.172089 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } attempt: 3
I20260620 02:34:21.173965 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.173348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34946)
0620 02:34:21.173637 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.173787 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 02:34:21.175079 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.174721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34956)
0620 02:34:21.174847 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.174926 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 02:34:21.175320 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39671: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
I20260620 02:34:21.318701 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } attempt: 4
I20260620 02:34:21.320122 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.319537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34964)
0620 02:34:21.319875 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.319988 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 02:34:21.321445 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.320991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34970)
0620 02:34:21.321175 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.321281 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:34:21.321694 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39671: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39671: connect: Connection refused (error 111)
W20260620 02:34:21.353128 22901 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:34:21.353528 22901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:21.353624 22901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:21.353749 22901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:21.361465 22901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:21.361577 22901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:21.361665 22901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:21.361765 22901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:21.361851 22901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:21.372480 22901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33825,127.0.97.61:39671,127.0.97.60:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:21.373946 22901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:21.375384 22901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:21.387868 22907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:21.387943 22906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:21.387918 22909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:21.389124 22901 server_base.cc:1034] running on GCE node
I20260620 02:34:21.390010 22901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:21.391636 22901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:21.392899 22901 hybrid_clock.cc:648] HybridClock initialized: now 1781922861392840 us; error 69 us; skew 500 ppm
I20260620 02:34:21.393285 22901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:21 dist-test-slave-r706 krb5kdc[22814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:21.396793 22901 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:34:21.398478 22901 webserver.cc:460] Webserver started at http://127.0.97.61:38309/ using document root <none> and password file <none>
I20260620 02:34:21.399278 22901 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:21.399382 22901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:21.399781 22901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:21.403048 22901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data/instance:
uuid: "c43a3b91214d439bacbdda53c6da9883"
format_stamp: "Formatted at 2026-06-20 02:34:21 on dist-test-slave-r706"
server_key: "99d773b676026090e25d34c13b58f5cf"
server_key_iv: "0f3b781a075eca9d777f245eec146144"
server_key_version: "encryptionkey@0"
I20260620 02:34:21.404073 22901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal/instance:
uuid: "c43a3b91214d439bacbdda53c6da9883"
format_stamp: "Formatted at 2026-06-20 02:34:21 on dist-test-slave-r706"
server_key: "99d773b676026090e25d34c13b58f5cf"
server_key_iv: "0f3b781a075eca9d777f245eec146144"
server_key_version: "encryptionkey@0"
I20260620 02:34:21.410385 22901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:34:21.414186 22916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:21.415558 22901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:34:21.415763 22901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal
uuid: "c43a3b91214d439bacbdda53c6da9883"
format_stamp: "Formatted at 2026-06-20 02:34:21 on dist-test-slave-r706"
server_key: "99d773b676026090e25d34c13b58f5cf"
server_key_iv: "0f3b781a075eca9d777f245eec146144"
server_key_version: "encryptionkey@0"
I20260620 02:34:21.416010 22901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:21.442394 22901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:21.447749 22901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:21.448109 22901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:21.483474 22901 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39671
I20260620 02:34:21.483494 22967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39671 every 8 connection(s)
I20260620 02:34:21.485095 22901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:34:21.489383 22968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:21.489580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22901
I20260620 02:34:21.489769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-1/wal/instance
I20260620 02:34:21.490207 388 external_mini_cluster.cc:1428] Setting key b3fd599c5c284abac8771eeb1172dfe5
I20260620 02:34:21.498386 22968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:21 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:34:21.525249 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.501216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.0.97.62:33825)
0620 02:34:21.501805 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:21.501812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:21.502898 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 02:34:21.506153 (+ 3255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:21.506158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:21.506237 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:21.506343 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:21.509113 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.509937 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.512762 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.513351 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.513469 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:21.518270 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:21.518276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:21.518280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:21.518331 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:21.521671 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:21.522582 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:21.522588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:21.522592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:21.522731 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:21.523193 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:21.523200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:21.523204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:21.523553 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:21.523732 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:21.524497 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 02:34:21.524832 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 02:34:21.526314 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.501767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33825 (local address 127.0.0.1:41874)
0620 02:34:21.502678 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.502741 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:21.505759 (+ 3018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:21.506669 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:21.506685 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:21.507653 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:21.508719 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:21.508755 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.510164 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.510179 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:21.512448 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:21.512471 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.513629 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.513643 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:21.514023 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:21.514867 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:21.514904 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:21.517956 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:21.521861 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:21.521872 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:21.521896 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:21.522436 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:21.522865 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:21.522870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:21.522874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:21.523073 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:21.523706 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:21.523714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:21.524113 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 02:34:21.524417 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 02:34:21.525028 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":327,"threads_started":1}
I20260620 02:34:21.528264 22968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:21 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:34:21.545840 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.529269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34972)
0620 02:34:21.529477 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.529503 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:21.529607 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:21.530730 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:21.530734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:21.531205 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:21.531343 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:21.531353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.532806 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.532814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:21.533737 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:21.533749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.534635 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.534640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:21.534824 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:21.535544 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:21.535574 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:21.538049 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:21.542943 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:21.542947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:21.542950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:21.543405 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:21.543937 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:21.543940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:21.543943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:21.544092 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:21.544897 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:21.544930 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:21.545164 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 02:34:21.545300 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 02:34:21.545568 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:34:21.546350 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.529531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.0.97.61:39671)
0620 02:34:21.530289 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:21.530296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:21.530325 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:34:21.530386 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:21.530393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:21.530578 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:21.530745 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:21.531528 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.532642 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.533922 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.534480 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.534626 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:21.538323 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:21.538344 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:21.538367 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:21.538415 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:21.542718 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:21.543588 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:21.543601 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:21.543607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:21.543761 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:21.544224 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:21.544232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:21.544237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:21.544752 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:21.545066 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:21.545522 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 02:34:21.545820 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":412,"threads_started":1}
I20260620 02:34:21.549970 22968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:21.551429 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.550816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53490)
0620 02:34:21.551077 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.551222 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 02:34:21.551744 22918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.553608 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.553222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53494)
0620 02:34:21.553408 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.553495 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 02:34:21.553876 22968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:34:21.569936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41735
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33825,127.0.97.61:39671,127.0.97.60:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:34:21.603237 22968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 1
I20260620 02:34:21.604498 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.604017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53504)
0620 02:34:21.604238 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.604332 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:34:21.605738 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.605291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53514)
0620 02:34:21.605478 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.605592 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:34:21.606025 22968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.621061 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } attempt: 5
Jun 20 02:34:21 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:34:21.641165 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.621877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34976)
0620 02:34:21.622082 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.622104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:21.622220 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:21.622671 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:21.622675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:21.623150 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:21.623332 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:21.623341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.625036 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.625044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:21.626225 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:21.626245 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.627963 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.627969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:21.628159 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:21.628923 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:21.628944 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:21.631961 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:21.638032 (+ 6071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:21.638038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:21.638041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:21.638823 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:21.639332 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:21.639336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:21.639339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:21.639481 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:21.640184 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:21.640189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:21.640341 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 02:34:21.640487 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:34:21.640794 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:34:21.641229 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.621969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.97.61:39671)
0620 02:34:21.622175 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:21.622181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:21.622207 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:34:21.622350 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:21.622358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:21.622468 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:21.622597 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:21.623509 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.624651 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.626571 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:21.627692 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:21.627895 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:21.632332 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:21.632343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:21.632351 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:21.632418 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:21.637752 (+ 5334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:21.639018 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:21.639026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:21.639029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:21.639209 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:21.639601 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:21.639607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:21.639611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:21.640021 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:21.640187 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:21.640624 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 02:34:21.640876 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:34:21.642601 22894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:21.643960 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.643496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53530)
0620 02:34:21.643690 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.643783 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 02:34:21.645285 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.644757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53542)
0620 02:34:21.644979 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.645134 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 02:34:21.645573 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.674355 22968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 2
I20260620 02:34:21.675673 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.675083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53548)
0620 02:34:21.675339 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.675456 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:34:21.676931 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.676516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53554)
0620 02:34:21.676687 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.676781 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:34:21.677353 22968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.696836 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 1
I20260620 02:34:21.698143 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.697655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53568)
0620 02:34:21.697857 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.697958 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 02:34:21.699299 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.698946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53576)
0620 02:34:21.699107 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.699185 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:34:21.699509 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.767738 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 2
I20260620 02:34:21.768610 22968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 3
I20260620 02:34:21.769086 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.768443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53578)
0620 02:34:21.768716 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.768829 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 02:34:21.769706 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.769244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53586)
0620 02:34:21.769423 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.769511 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 02:34:21.770174 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.769755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53598)
0620 02:34:21.769966 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.770047 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 02:34:21.770583 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.770792 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.770322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53612)
0620 02:34:21.770568 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.770658 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 02:34:21.771005 22968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.876839 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 3
I20260620 02:34:21.878201 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.877699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53620)
0620 02:34:21.877930 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.878023 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:34:21.879408 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.879038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53628)
0620 02:34:21.879216 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.879300 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:34:21.879729 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:21.914263 22968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 4
I20260620 02:34:21.915575 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.914984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53642)
0620 02:34:21.915233 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.915356 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 02:34:21.916781 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:21.916382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53650)
0620 02:34:21.916570 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:21.916654 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:34:21.917064 22968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
W20260620 02:34:21.957752 22976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:34:21.958485 22976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:21.958734 22976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:21.958971 22976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:21.973651 22976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:21.973852 22976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:21.974045 22976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:21.974237 22976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:21.974460 22976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:21.987285 22976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33825,127.0.97.61:39671,127.0.97.60:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:21.988758 22976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:21.990222 22976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:22.004094 22982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:22.004115 22984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:22.003931 22981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:22.004743 22976 server_base.cc:1034] running on GCE node
I20260620 02:34:22.005717 22976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:22.007225 22976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:22.008481 22976 hybrid_clock.cc:648] HybridClock initialized: now 1781922862008410 us; error 38 us; skew 500 ppm
I20260620 02:34:22.008872 22976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:22.012334 22976 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:34:22.014019 22976 webserver.cc:460] Webserver started at http://127.0.97.60:45003/ using document root <none> and password file <none>
I20260620 02:34:22.014780 22976 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:22.014891 22976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:22.015185 22976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:22.018642 22976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data/instance:
uuid: "21b3955801a548fa9b614f2f1e6f72b8"
format_stamp: "Formatted at 2026-06-20 02:34:22 on dist-test-slave-r706"
server_key: "af98add5e33a247354cbeb933e04863d"
server_key_iv: "c2e423f6821ace9b40624f34e8be2913"
server_key_version: "encryptionkey@0"
I20260620 02:34:22.019515 22976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal/instance:
uuid: "21b3955801a548fa9b614f2f1e6f72b8"
format_stamp: "Formatted at 2026-06-20 02:34:22 on dist-test-slave-r706"
server_key: "af98add5e33a247354cbeb933e04863d"
server_key_iv: "c2e423f6821ace9b40624f34e8be2913"
server_key_version: "encryptionkey@0"
I20260620 02:34:22.026170 22976 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:34:22.031183 22991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:22.032820 22976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:34:22.033061 22976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal
uuid: "21b3955801a548fa9b614f2f1e6f72b8"
format_stamp: "Formatted at 2026-06-20 02:34:22 on dist-test-slave-r706"
server_key: "af98add5e33a247354cbeb933e04863d"
server_key_iv: "c2e423f6821ace9b40624f34e8be2913"
server_key_version: "encryptionkey@0"
I20260620 02:34:22.033286 22976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:22.053187 22976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:22.056973 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 4
I20260620 02:34:22.058737 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.057693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53662)
0620 02:34:22.058309 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.058429 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 02:34:22.059604 22976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:22.060379 22976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:22.060668 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.060059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53676)
0620 02:34:22.060384 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.060485 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260620 02:34:22.060962 22894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41735: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41735: connect: Connection refused (error 111)
I20260620 02:34:22.085206 22976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41735
I20260620 02:34:22.085255 23042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41735 every 8 connection(s)
I20260620 02:34:22.086695 22976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:34:22.090631 23043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:22.093358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 22976
I20260620 02:34:22.093559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922485472743-388-0/minicluster-data/master-2/wal/instance
I20260620 02:34:22.094015 388 external_mini_cluster.cc:1428] Setting key 85b287ffc9100e597ee1c1b9142eac17
I20260620 02:34:22.098752 23043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:22.118593 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.101837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.0.97.62:33825)
0620 02:34:22.102471 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.102479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.102941 (+ 462us) server_negotiation.cc:407] Connection header received
0620 02:34:22.103207 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.103212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.103286 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.103421 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.104650 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.105510 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.107038 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.107471 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.107619 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.111682 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.111688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.111691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.111729 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.115026 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.116010 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.116016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.116020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.116182 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.116630 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.116634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.116636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.117099 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.117261 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.117849 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.118284 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":299,"threads_started":1}
I20260620 02:34:22.119794 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.096033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.0.97.60:41735)
0620 02:34:22.096898 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.096910 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.096956 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:34:22.098735 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.098761 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.099185 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.099554 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.101584 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.103551 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.104963 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.105780 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.106004 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.111096 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.111110 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.111129 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.111170 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.114342 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.115241 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.115250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.115253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.115541 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.116049 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.116052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.116054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.116794 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.117124 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.117707 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.118202 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"spinlock_wait_cycles":24448,"thread_start_us":410,"threads_started":1}
I20260620 02:34:22.119813 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.101868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33825 (local address 127.0.0.1:41876)
0620 02:34:22.102783 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.102830 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:22.103050 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:22.103484 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:22.103491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:22.104216 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:22.104486 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.104496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.105687 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.105692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.106861 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.106874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.107722 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.107731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.107890 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.108581 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:22.108605 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:22.111405 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:22.115222 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.115232 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.115246 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.115866 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.116304 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.116308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.116311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.116482 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.117229 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:22.117235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:22.117499 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 02:34:22.117707 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 02:34:22.118308 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":172,"thread_start_us":521,"threads_started":1}
I20260620 02:34:22.122582 23043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:22.138057 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.123420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.0.97.61:39671)
0620 02:34:22.123634 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.123640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.123802 (+ 162us) server_negotiation.cc:407] Connection header received
0620 02:34:22.123994 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.123999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.124064 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.124176 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.124912 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.125686 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.126955 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.127392 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.127533 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.131181 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.131187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.131191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.131229 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.134654 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.135770 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.135775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.135777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.135947 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.136430 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.136435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.136439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.136833 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.137004 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.137432 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.137660 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:34:22.138154 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.123466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39671 (local address 127.0.0.1:34988)
0620 02:34:22.123708 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.123723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:22.123867 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:22.124198 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:22.124203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:22.124607 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:22.124763 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.124772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.125862 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.125868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.126782 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.126794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.127547 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.127553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.127758 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.128415 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:22.128444 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:22.130881 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:22.134966 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.134975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.134981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.135602 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.136103 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.136109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.136112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.136293 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.136965 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:22.136971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:22.137176 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 02:34:22.137334 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 02:34:22.137671 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260620 02:34:22.139257 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.123333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.0.97.62:33825)
0620 02:34:22.123724 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.123732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.123766 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:34:22.123823 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.123827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.123913 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.124024 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.125336 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.126052 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.127266 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.127711 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.127860 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.132179 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.132185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.132188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.132225 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.135727 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.136585 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.136592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.136596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.136783 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.137329 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.137335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.137357 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.137726 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.138048 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.138647 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.138937 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:34:22.139243 23043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:34:22.154357 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.140013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53694)
0620 02:34:22.140179 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.140202 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:22.140310 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:22.140587 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:22.140592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:22.141161 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:22.141366 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.141382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.142521 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.142529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.143682 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.143695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.144542 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.144549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.144771 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.145467 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:22.145494 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:22.148261 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:22.151808 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.151812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.151815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.152238 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.152684 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.152688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.152690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.152818 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.153509 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:22.153517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:22.153689 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 02:34:22.153819 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:34:22.154082 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 02:34:22.154655 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.140062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.0.97.60:41735)
0620 02:34:22.140329 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.140333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.140348 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:34:22.140397 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.140400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.140454 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.140529 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.141540 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.142364 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.143836 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.144374 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.144473 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.148516 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.148522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.148525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.148556 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.151674 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.152377 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.152381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.152383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.152569 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.152929 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.152934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.152937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.153379 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.153549 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.153954 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.154270 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":168}
I20260620 02:34:22.158159 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.140968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.97.61:39671)
0620 02:34:22.141154 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.141158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.141291 (+ 133us) server_negotiation.cc:407] Connection header received
0620 02:34:22.141411 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.141415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.141538 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.141656 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.143165 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.143984 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.145738 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.146188 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.146294 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.151731 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.151741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.151749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.151802 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.155260 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.156135 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.156140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.156144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.156318 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.156705 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.156710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.156713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.157107 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.157380 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.157596 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.157822 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 02:34:22.160676 23043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: Bootstrap starting.
I20260620 02:34:22.163115 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:22.164682 23043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:22.166157 23043 log.cc:824] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:22.169473 23043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: No bootstrap required, opened a new log
I20260620 02:34:22.166849 22869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9793cc81eb254b008e6a44278b8dc367"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41892
I20260620 02:34:22.177445 23043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.177810 23043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b3955801a548fa9b614f2f1e6f72b8, State: Initialized, Role: FOLLOWER
I20260620 02:34:22.178313 23043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.179394 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.179852 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.180462 23043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:34:22.190305 23061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:34:22.190495 23061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:34:22.188231 22976 master_runner.cc:186] Error getting master registration for 127.0.97.62:33825: OK, instance_id { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" instance_seqno: 1781922860906731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:34:22.196650 22976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:34:22.216297 22968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 5
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:34:22.233105 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.217373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.0.97.60:41735)
0620 02:34:22.217589 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.217598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.217623 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:34:22.217689 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.217694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.217774 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.217879 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.218923 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.220112 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.221953 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.222737 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.222869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.227042 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.227053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.227057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.227091 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.230092 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.230824 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.230841 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.230845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.230974 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.231381 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.231387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.231390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.231899 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.232065 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.232467 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.232693 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:34:22.233096 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.217141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53708)
0620 02:34:22.217334 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.217354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:22.217502 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:22.217960 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:22.217966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:22.218524 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:22.218748 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.218761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.220282 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.220288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.221788 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.221804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.222917 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.222923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.223135 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.223938 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:22.223964 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:22.226819 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:22.230256 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.230262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.230267 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.230688 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.231092 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.231098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.231102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.231252 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.232021 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:22.232026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:22.232188 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:34:22.232344 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:34:22.232667 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:34:22.239760 22968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: Bootstrap starting.
I20260620 02:34:22.244191 22968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:22.245658 22968 log.cc:824] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:22.249306 22968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: No bootstrap required, opened a new log
I20260620 02:34:22.260932 22968 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.261436 22968 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43a3b91214d439bacbdda53c6da9883, State: Initialized, Role: FOLLOWER
I20260620 02:34:22.262029 22968 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.263202 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.263635 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.264509 22968 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:34:22.275547 23075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:34:22.275730 23075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:34:22.272315 22901 master_runner.cc:186] Error getting master registration for 127.0.97.62:33825: OK, instance_id { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" instance_seqno: 1781922860906731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:34:22.284411 22901 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:34:22.314937 22997 tablet.cc:2369] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:34:22.338397 22894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } attempt: 5
Jun 20 02:34:22 dist-test-slave-r706 krb5kdc[22814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:34:22.357239 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.339199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41735 (local address 127.0.0.1:53722)
0620 02:34:22.339493 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:22.339514 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:22.339653 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:22.340106 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:22.340114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:22.340682 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:22.340926 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.340939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.342504 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.342509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.344063 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:22.344080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.345210 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.345217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:22.345416 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.346183 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:22.346211 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:22.349279 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:22.354018 (+ 4739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.354025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.354030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.354631 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.355125 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:22.355132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:22.355135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:22.355347 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:22.356097 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:22.356102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:22.356277 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 02:34:22.356433 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:34:22.356786 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 02:34:22.358376 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.339265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.0.97.60:41735)
0620 02:34:22.339495 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.339502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.339580 (+ 78us) server_negotiation.cc:407] Connection header received
0620 02:34:22.339827 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.339839 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.339945 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.340055 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.341135 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.342311 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.344247 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.345038 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.345183 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.349587 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.349596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.349601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.349648 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.353801 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.354802 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.354808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.354812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.354982 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.355497 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.355503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.355507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.355877 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.356028 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.357760 (+ 1732us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.358012 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:34:22.364965 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367: Bootstrap starting.
I20260620 02:34:22.369424 22894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:22.370859 22894 log.cc:824] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:22.374802 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367: No bootstrap required, opened a new log
I20260620 02:34:22.383044 22894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.383920 22894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793cc81eb254b008e6a44278b8dc367, State: Initialized, Role: FOLLOWER
I20260620 02:34:22.384502 22894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.390013 23078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.390877 23078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.391225 22894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:22.391296 22869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:34:22.391503 22869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:34:22.395273 22848 tablet.cc:2369] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:34:22.395267 22869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.397204 22869 leader_election.cc:288] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [CANDIDATE]: Term 1 election: Requested vote from peers c43a3b91214d439bacbdda53c6da9883 (127.0.97.61:39671), 21b3955801a548fa9b614f2f1e6f72b8 (127.0.97.60:41735)
I20260620 02:34:22.398674 22943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9793cc81eb254b008e6a44278b8dc367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c43a3b91214d439bacbdda53c6da9883"
I20260620 02:34:22.399228 22943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:22.401885 23018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9793cc81eb254b008e6a44278b8dc367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21b3955801a548fa9b614f2f1e6f72b8"
I20260620 02:34:22.402429 23018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:22.404600 22943 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793cc81eb254b008e6a44278b8dc367 in term 1.
I20260620 02:34:22.405372 22844 leader_election.cc:302] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793cc81eb254b008e6a44278b8dc367, c43a3b91214d439bacbdda53c6da9883; no voters:
I20260620 02:34:22.406328 23018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793cc81eb254b008e6a44278b8dc367 in term 1.
I20260620 02:34:22.410777 23078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:22.420814 23078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [term 1 LEADER]: Becoming Leader. State: Replica: 9793cc81eb254b008e6a44278b8dc367, State: Running, Role: LEADER
I20260620 02:34:22.421823 23078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } }
I20260620 02:34:22.432718 23082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9793cc81eb254b008e6a44278b8dc367. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.433320 23082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:22.438458 23087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 02:34:22.451153 22922 tablet.cc:2369] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:34:22.464763 23087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:34:22.536809 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.444113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.0.97.61:39671)
0620 02:34:22.444291 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.444298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.455784 (+ 11486us) server_negotiation.cc:407] Connection header received
0620 02:34:22.456074 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.456082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.456170 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.456245 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.474584 (+ 18339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.475765 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.504822 (+ 29057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.505614 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.505739 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.517637 (+ 11898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.517646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.517651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.517704 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.521774 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.522966 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.522972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.522980 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.523158 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.533516 (+ 10358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.533525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.533531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.534043 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.534358 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.536075 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.536394 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:34:22.538923 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.443150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.0.97.62:33825)
0620 02:34:22.443341 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.443349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.454035 (+ 10686us) server_negotiation.cc:407] Connection header received
0620 02:34:22.465351 (+ 11316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.465358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.465455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.465546 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.468066 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.469305 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.504593 (+ 35288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.524567 (+ 19974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.524716 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.531079 (+ 6363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.531087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.531092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.531143 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.535080 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.536065 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.536072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.536076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.536236 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.536710 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.536715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.536719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.537264 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.537576 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.538176 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.538491 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:34:22.542546 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:22.454760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.0.97.60:41735)
0620 02:34:22.455001 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.455008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.470003 (+ 14995us) server_negotiation.cc:407] Connection header received
0620 02:34:22.470277 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.470285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.470370 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.470445 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.501208 (+ 30763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.502401 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.509677 (+ 7276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:22.510478 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:22.510613 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:22.523752 (+ 13139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:22.523762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:22.523766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:22.523816 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:22.528172 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.539274 (+ 11102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.539284 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.539287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.539496 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:22.540522 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:22.540527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:22.540530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:22.540969 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:22.541261 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:22.541805 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 02:34:22.542114 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:34:22.543457 22943 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [term 1 FOLLOWER]: Refusing update from remote peer 9793cc81eb254b008e6a44278b8dc367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:34:22.545125 23018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [term 1 FOLLOWER]: Refusing update from remote peer 9793cc81eb254b008e6a44278b8dc367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:34:22.546558 23078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:34:22.548105 23078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:34:22.592900 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9793cc81eb254b008e6a44278b8dc367. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.593317 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.598096 23105 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:34:22.598629 23098 mvcc.cc:205] Tried to move back new op lower bound from 7298756044962942976 to 7298756044500025344. Current Snapshot: MvccSnapshot[applied={T|T < 7298756044962942976}]
I20260620 02:34:22.605255 23102 mvcc.cc:205] Tried to move back new op lower bound from 7298756044962942976 to 7298756044500025344. Current Snapshot: MvccSnapshot[applied={T|T < 7298756044962942976}]
I20260620 02:34:22.606901 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9793cc81eb254b008e6a44278b8dc367. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.607303 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.615105 23101 mvcc.cc:205] Tried to move back new op lower bound from 7298756044962942976 to 7298756044500025344. Current Snapshot: MvccSnapshot[applied={T|T < 7298756044962942976}]
I20260620 02:34:22.617107 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.617405 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.621447 23087 catalog_manager.cc:1347] Generated new cluster ID: 9eeeedd7b73a450c92bfb0f9915a45dc
I20260620 02:34:22.621573 23087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:22.631497 23082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.631847 23082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:22.632273 23082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.632769 23082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:22.639914 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9793cc81eb254b008e6a44278b8dc367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793cc81eb254b008e6a44278b8dc367" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33825 } } peers { permanent_uuid: "c43a3b91214d439bacbdda53c6da9883" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39671 } } peers { permanent_uuid: "21b3955801a548fa9b614f2f1e6f72b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41735 } } }
I20260620 02:34:22.640208 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:22.671013 23087 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:22.673048 23087 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:22.696540 23087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9793cc81eb254b008e6a44278b8dc367: Generated new TSK 0
I20260620 02:34:22.697842 23087 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:34:22.844770 388 mini_kdc.cc:348] Time spent kdestroy: real 0.124s user 0.000s sys 0.046s
W20260620 02:34:22.865875 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.849748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.0.97.62:33825)
0620 02:34:22.850174 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.850180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.862417 (+ 12237us) server_negotiation.cc:407] Connection header received
0620 02:34:22.863910 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.863917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.864016 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.864103 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.865424 (+ 1321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41912: BlockingRecv error: recv got EOF from 127.0.0.1:41912 (error 108)
Metrics: {"server-negotiator.queue_time_us":260}
W20260620 02:34:22.870337 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.849524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.0.97.61:39671)
0620 02:34:22.849969 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.849977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.867587 (+ 17610us) server_negotiation.cc:407] Connection header received
0620 02:34:22.868299 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.868306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.868428 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.868518 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.870026 (+ 1508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35016: BlockingRecv error: recv got EOF from 127.0.0.1:35016 (error 108)
Metrics: {"server-negotiator.queue_time_us":296}
W20260620 02:34:22.870395 23046 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.863059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.0.97.60:41735)
0620 02:34:22.863354 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.863365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.867958 (+ 4593us) server_negotiation.cc:407] Connection header received
0620 02:34:22.868511 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.868519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.868640 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.868729 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.870127 (+ 1398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53730: BlockingRecv error: recv got EOF from 127.0.0.1:53730 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:34:22.881052 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.878017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.97.61:39671)
0620 02:34:22.878266 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.878274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.878792 (+ 518us) server_negotiation.cc:407] Connection header received
0620 02:34:22.879169 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.879176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.879338 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.879433 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.880684 (+ 1251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35032: BlockingRecv error: recv got EOF from 127.0.0.1:35032 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260620 02:34:22.888973 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.886596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.97.61:39671)
0620 02:34:22.886775 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.886781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.887249 (+ 468us) server_negotiation.cc:407] Connection header received
0620 02:34:22.887537 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.887543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.887645 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.887722 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.888716 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35044: BlockingRecv error: recv got EOF from 127.0.0.1:35044 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 02:34:22.899313 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.896384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.0.97.61:39671)
0620 02:34:22.896594 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.896602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.897316 (+ 714us) server_negotiation.cc:407] Connection header received
0620 02:34:22.897656 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.897663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.897786 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.897876 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.898992 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35054: BlockingRecv error: recv got EOF from 127.0.0.1:35054 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:34:22.911278 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.908772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.0.97.61:39671)
0620 02:34:22.909087 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.909094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.909643 (+ 549us) server_negotiation.cc:407] Connection header received
0620 02:34:22.909866 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.909872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.909981 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.910100 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.911031 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35066: BlockingRecv error: recv got EOF from 127.0.0.1:35066 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
W20260620 02:34:22.945560 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.943153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.97.61:39671)
0620 02:34:22.943390 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.943397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.943954 (+ 557us) server_negotiation.cc:407] Connection header received
0620 02:34:22.944166 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.944173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.944271 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.944409 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.945298 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35082: BlockingRecv error: recv got EOF from 127.0.0.1:35082 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:34:22.974730 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:22.972318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.97.61:39671)
0620 02:34:22.972670 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:22.972677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:22.973272 (+ 595us) server_negotiation.cc:407] Connection header received
0620 02:34:22.973522 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:22.973526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:22.973610 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:22.973695 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:22.974539 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35090: BlockingRecv error: recv got EOF from 127.0.0.1:35090 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:34:23.013111 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:23.010773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.97.61:39671)
0620 02:34:23.010966 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.010971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.011390 (+ 419us) server_negotiation.cc:407] Connection header received
0620 02:34:23.011603 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.011607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.011683 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.011774 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:23.012768 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35104: BlockingRecv error: recv got EOF from 127.0.0.1:35104 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 02:34:23.083884 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:23.081370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.97.61:39671)
0620 02:34:23.081661 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.081668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.081983 (+ 315us) server_negotiation.cc:407] Connection header received
0620 02:34:23.082236 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.082243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.082354 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.082466 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:23.083629 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:34:23.194284 23061 catalog_manager.cc:1259] Loaded cluster ID: 9eeeedd7b73a450c92bfb0f9915a45dc
I20260620 02:34:23.194432 23061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: loading cluster ID for follower catalog manager: success
I20260620 02:34:23.198408 23061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: acquiring CA information for follower catalog manager: success
I20260620 02:34:23.200852 23061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21b3955801a548fa9b614f2f1e6f72b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:34:23.223186 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:23.221158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.97.61:39671)
0620 02:34:23.221411 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.221421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.221779 (+ 358us) server_negotiation.cc:407] Connection header received
0620 02:34:23.221946 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.221950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.222025 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.222117 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:23.222932 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35110: BlockingRecv error: recv got EOF from 127.0.0.1:35110 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:34:23.278077 23075 catalog_manager.cc:1259] Loaded cluster ID: 9eeeedd7b73a450c92bfb0f9915a45dc
I20260620 02:34:23.278232 23075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: loading cluster ID for follower catalog manager: success
I20260620 02:34:23.280838 23075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: acquiring CA information for follower catalog manager: success
I20260620 02:34:23.282900 23075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c43a3b91214d439bacbdda53c6da9883: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:34:23.490056 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:23.483000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.97.61:39671)
0620 02:34:23.483211 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.483219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.483465 (+ 246us) server_negotiation.cc:407] Connection header received
0620 02:34:23.483706 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.483717 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.483921 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.484052 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:23.484460 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.485669 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.487142 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.487821 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.487950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:23.488506 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:23.489233 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:23.489378 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:23.489610 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 02:34:23.489707 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:34:23.506198 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:23.498324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.0.97.60:41735)
0620 02:34:23.499484 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.499490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.499509 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:34:23.499624 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.499629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.499835 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.500021 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:23.500676 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.501858 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.503845 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.504252 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.504356 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:23.504887 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:23.505548 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:23.505660 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:23.505783 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 02:34:23.505856 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":934,"threads_started":1}
I20260620 02:34:23.520614 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:23.512761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.0.97.61:39671)
0620 02:34:23.512990 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.513022 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.513853 (+ 831us) server_negotiation.cc:407] Connection header received
0620 02:34:23.514148 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.514155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.514249 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.514324 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:23.515235 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.516268 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.518552 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.519086 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.519187 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:23.519774 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:23.519916 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:23.520008 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:23.520213 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 02:34:23.520305 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 02:34:23.520673 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:23.514078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.0.97.60:41735)
0620 02:34:23.514252 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.514257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.514847 (+ 590us) server_negotiation.cc:407] Connection header received
0620 02:34:23.515190 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.515197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.515286 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.515402 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:23.515903 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.516690 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.518759 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.519205 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.519335 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:23.519856 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:23.519978 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:23.520077 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:23.520275 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 02:34:23.520372 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:34:23.522418 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:23.512588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.0.97.62:33825)
0620 02:34:23.513482 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:23.513488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:23.513510 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:34:23.513667 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:23.513675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:23.513913 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:23.514163 (+ 250us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:23.515042 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.516038 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.519311 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:23.520008 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:23.520165 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:23.520899 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:23.521422 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:23.521517 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:23.521866 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 02:34:23.521962 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":604,"threads_started":1}
I20260620 02:34:23.528692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22827
I20260620 02:34:23.529935 22889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:23.642124 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22827
W20260620 02:34:23.665727 22993 connection.cc:535] server connection from 127.0.0.1:53722 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:34:23.666180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22901
I20260620 02:34:23.667408 22963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:23.777137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22901
I20260620 02:34:23.800155 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 22976
I20260620 02:34:23.801499 23038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:23.912487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 22976
2026-06-20T02:34:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4964 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:34:24 dist-test-slave-r706 krb5kdc[23187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:34:24 dist-test-slave-r706 krb5kdc[23187](info): set up 2 sockets
Jun 20 02:34:24 dist-test-slave-r706 krb5kdc[23187](info): commencing operation
krb5kdc: starting...
W20260620 02:34:25.049850 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/test-user.keytab.
Jun 20 02:34:25 dist-test-slave-r706 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:34:25.374420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43191
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43191,127.0.97.61:37613,127.0.97.60:42885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:34:25.729630 23200 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:34:25.730111 23200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:25.730258 23200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:25.730405 23200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:25.738168 23200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:25.738336 23200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:25.738451 23200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:25.738548 23200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:25.738670 23200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:25.749162 23200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43191,127.0.97.61:37613,127.0.97.60:42885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:25.750643 23200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:25.752065 23200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:25.765128 23205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:25.765059 23208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:25.765139 23206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:25.766239 23200 server_base.cc:1034] running on GCE node
I20260620 02:34:25.767086 23200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:25.768522 23200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:25.769749 23200 hybrid_clock.cc:648] HybridClock initialized: now 1781922865769690 us; error 38 us; skew 500 ppm
I20260620 02:34:25.770118 23200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:25 dist-test-slave-r706 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:25.773481 23200 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:34:25.775058 23200 webserver.cc:460] Webserver started at http://127.0.97.62:37491/ using document root <none> and password file <none>
I20260620 02:34:25.775791 23200 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:25.775882 23200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:25.776276 23200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:25.779686 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data/instance:
uuid: "7e8a60125c004ba797ef60f9be570963"
format_stamp: "Formatted at 2026-06-20 02:34:25 on dist-test-slave-r706"
server_key: "aa79422eb95b723ed0d2d1b4de961b28"
server_key_iv: "bb75a9de1d6a983a1964954e1e6a8b4f"
server_key_version: "encryptionkey@0"
I20260620 02:34:25.780616 23200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal/instance:
uuid: "7e8a60125c004ba797ef60f9be570963"
format_stamp: "Formatted at 2026-06-20 02:34:25 on dist-test-slave-r706"
server_key: "aa79422eb95b723ed0d2d1b4de961b28"
server_key_iv: "bb75a9de1d6a983a1964954e1e6a8b4f"
server_key_version: "encryptionkey@0"
I20260620 02:34:25.786815 23200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 02:34:25.790807 23215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:25.792294 23200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:34:25.792516 23200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal
uuid: "7e8a60125c004ba797ef60f9be570963"
format_stamp: "Formatted at 2026-06-20 02:34:25 on dist-test-slave-r706"
server_key: "aa79422eb95b723ed0d2d1b4de961b28"
server_key_iv: "bb75a9de1d6a983a1964954e1e6a8b4f"
server_key_version: "encryptionkey@0"
I20260620 02:34:25.792766 23200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:25.810468 23200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:25.815261 23200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:25.815577 23200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:25.835279 23200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43191
I20260620 02:34:25.835299 23266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43191 every 8 connection(s)
I20260620 02:34:25.836694 23200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:34:25.840590 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:25.846182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 23200
I20260620 02:34:25.846405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-0/wal/instance
I20260620 02:34:25.846868 388 external_mini_cluster.cc:1428] Setting key 8053680493715814faf8fb9ef4bc3102
I20260620 02:34:25.847600 23267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 02:34:25 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:25.892210 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:25.850974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43191 (local address 127.0.0.1:53902)
0620 02:34:25.862577 (+ 11603us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:25.862652 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:25.864789 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:25.865998 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:25.866004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:25.866601 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:25.866964 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:25.866981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:25.869451 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:25.869455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:25.870583 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:25.870597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:25.872078 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:25.872082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:25.872204 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:25.880648 (+ 8444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:25.880680 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:25.884124 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:25.887766 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:25.887778 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:25.887791 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:25.888192 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:25.888625 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:25.888633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:25.888638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:25.888807 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:25.889673 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:25.889686 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:25.889924 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 02:34:25.890187 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 02:34:25.890795 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11265,"mutex_wait_us":137,"spinlock_wait_cycles":29952,"thread_start_us":11054,"threads_started":1}
I20260620 02:34:25.892167 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:25.850974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.97.62:43191)
0620 02:34:25.861862 (+ 10888us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:25.861883 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:25.862850 (+ 967us) server_negotiation.cc:407] Connection header received
0620 02:34:25.865301 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:25.865313 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:25.865721 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:25.865955 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:25.867205 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:25.869272 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:25.870761 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:25.871827 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:25.872052 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:25.884413 (+ 12361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:25.884422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:25.884437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:25.884505 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:25.887593 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:25.888337 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:25.888353 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:25.888356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:25.888504 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:25.888934 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:25.888939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:25.888944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:25.889528 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:25.889754 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:25.890317 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 02:34:25.890667 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":10622,"thread_start_us":10368,"threads_started":1}
I20260620 02:34:25.896013 23267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:25.897322 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:25.896856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52610)
0620 02:34:25.897061 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:25.897197 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:34:25.897637 23218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
I20260620 02:34:25.899374 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:25.898980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52618)
0620 02:34:25.899158 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:25.899243 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:34:25.899590 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37613: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:34:25.920907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37613
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43191,127.0.97.61:37613,127.0.97.60:42885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:34:25.948946 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } attempt: 1
I20260620 02:34:25.950220 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:25.949762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52626)
0620 02:34:25.949970 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:25.950089 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:34:25.951300 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:25.950941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52630)
0620 02:34:25.951098 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:25.951174 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:34:25.951501 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37613: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
I20260620 02:34:26.019831 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } attempt: 2
I20260620 02:34:26.021196 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.020583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52642)
0620 02:34:26.020875 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.020979 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 02:34:26.022339 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.021945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52648)
0620 02:34:26.022134 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.022215 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 02:34:26.022554 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37613: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
I20260620 02:34:26.113863 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } attempt: 3
I20260620 02:34:26.115134 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.114658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52652)
0620 02:34:26.114878 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.114979 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 02:34:26.116343 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.115844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52662)
0620 02:34:26.116060 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.116166 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 02:34:26.116647 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37613: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
I20260620 02:34:26.259971 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } attempt: 4
I20260620 02:34:26.261279 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.260691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52674)
0620 02:34:26.261060 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.261147 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 02:34:26.262455 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.261975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52676)
0620 02:34:26.262177 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.262287 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 02:34:26.262733 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37613: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37613: connect: Connection refused (error 111)
W20260620 02:34:26.272323 23274 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:34:26.272742 23274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:26.272871 23274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:26.273066 23274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:26.280788 23274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:26.280956 23274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:26.281076 23274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:26.281167 23274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:26.281263 23274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:26.291658 23274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43191,127.0.97.61:37613,127.0.97.60:42885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:26.292986 23274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:26.294284 23274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:26.306615 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:26.306710 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:26.306631 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:26.307600 23274 server_base.cc:1034] running on GCE node
I20260620 02:34:26.308408 23274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:26.309965 23274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:26.311219 23274 hybrid_clock.cc:648] HybridClock initialized: now 1781922866311147 us; error 48 us; skew 500 ppm
I20260620 02:34:26.311643 23274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:26.315094 23274 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:34:26.316794 23274 webserver.cc:460] Webserver started at http://127.0.97.61:43013/ using document root <none> and password file <none>
I20260620 02:34:26.317610 23274 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:26.317759 23274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:26.318110 23274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:26.321262 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data/instance:
uuid: "aff52d32fbac4223acfde1757bcc3d70"
format_stamp: "Formatted at 2026-06-20 02:34:26 on dist-test-slave-r706"
server_key: "ceba9ac962d9333858a1e75691c2b85b"
server_key_iv: "f2adb7fa7aa3d01832037e8ee7b956f4"
server_key_version: "encryptionkey@0"
I20260620 02:34:26.322106 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal/instance:
uuid: "aff52d32fbac4223acfde1757bcc3d70"
format_stamp: "Formatted at 2026-06-20 02:34:26 on dist-test-slave-r706"
server_key: "ceba9ac962d9333858a1e75691c2b85b"
server_key_iv: "f2adb7fa7aa3d01832037e8ee7b956f4"
server_key_version: "encryptionkey@0"
I20260620 02:34:26.327544 23274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:34:26.331166 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:26.332542 23274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:34:26.332749 23274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal
uuid: "aff52d32fbac4223acfde1757bcc3d70"
format_stamp: "Formatted at 2026-06-20 02:34:26 on dist-test-slave-r706"
server_key: "ceba9ac962d9333858a1e75691c2b85b"
server_key_iv: "f2adb7fa7aa3d01832037e8ee7b956f4"
server_key_version: "encryptionkey@0"
I20260620 02:34:26.333057 23274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:26.360212 23274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:26.364475 23274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:26.364780 23274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:26.384135 23274 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37613
I20260620 02:34:26.384155 23340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37613 every 8 connection(s)
I20260620 02:34:26.385597 23274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:34:26.389437 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:26.393613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 23274
I20260620 02:34:26.393824 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-1/wal/instance
I20260620 02:34:26.394359 388 external_mini_cluster.cc:1428] Setting key e490b0e348f31912728bcd7cbbe89271
I20260620 02:34:26.398275 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:26.423830 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.401808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.97.62:43191)
0620 02:34:26.402374 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.402382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.403511 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 02:34:26.405564 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.405572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.405681 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.405819 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.407695 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.408461 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.411139 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.411589 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.411700 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.416921 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:26.416928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:26.416931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:26.416982 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:26.419900 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.421072 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.421077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.421079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.421207 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.421763 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.421768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.421771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.422113 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:26.422230 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:26.423362 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0620 02:34:26.423555 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":293,"threads_started":1}
I20260620 02:34:26.425379 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.402457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43191 (local address 127.0.0.1:53908)
0620 02:34:26.403293 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.403347 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:26.405261 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:26.406086 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:26.406095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:26.406700 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:26.407353 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.407381 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.408651 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.408664 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.410894 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.410914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.411853 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.411865 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.412228 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.413071 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:26.413107 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:26.416580 (+ 3473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:26.420084 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.420099 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.420122 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.420872 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.421326 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.421334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.421340 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.421620 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.422260 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:26.422272 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:26.422826 (+ 554us) client_negotiation.cc:769] Sending connection context
0620 02:34:26.423195 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 02:34:26.423838 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":369,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:34:26.427214 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:34:26.445515 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.428126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52688)
0620 02:34:26.428350 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.428373 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:26.428462 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:26.429690 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:26.429698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:26.430109 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:26.430257 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.430266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.432471 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.432475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.433390 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.433401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.434637 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.434648 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.434834 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.435485 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:26.435502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:26.437952 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:26.442250 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.442258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.442264 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.442882 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.443285 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.443293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.443299 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.443464 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.444228 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:26.444237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:26.444441 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 02:34:26.444606 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 02:34:26.444969 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":38}
I20260620 02:34:26.445546 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.428328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.97.61:37613)
0620 02:34:26.429158 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.429165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.429199 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:34:26.429242 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.429273 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.429478 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.429650 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.430458 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.432280 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.433532 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.434472 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.434619 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.438183 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:26.438200 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:26.438221 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:26.438268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:26.442086 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.443046 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.443054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.443057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.443171 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.443602 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.443606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.443608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.443993 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:26.444296 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:26.444809 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 02:34:26.445182 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":619,"thread_start_us":429,"threads_started":1}
I20260620 02:34:26.448423 23341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:26.449846 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.449301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48306)
0620 02:34:26.449540 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.449684 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:34:26.450136 23290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.451851 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.451467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48322)
0620 02:34:26.451642 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.451731 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 02:34:26.452062 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:34:26.473683 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkyewzx/build/asan/bin/kudu
/tmp/dist-test-taskkyewzx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42885
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43191,127.0.97.61:37613,127.0.97.60:42885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:34:26.501369 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 1
I20260620 02:34:26.502743 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.502093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48334)
0620 02:34:26.502359 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.502506 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 02:34:26.504200 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.503696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48336)
0620 02:34:26.503915 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.504029 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 02:34:26.504539 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.562081 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } attempt: 5
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:34:26.572774 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 2
I20260620 02:34:26.573858 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.573410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48352)
0620 02:34:26.573612 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.573696 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:34:26.575142 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.574764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48368)
0620 02:34:26.574925 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.575006 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:34:26.575433 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.578594 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.563003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52704)
0620 02:34:26.563323 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.563344 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:26.563499 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:26.563933 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:26.563938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:26.564451 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:26.564698 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.564709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.565805 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.565812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.567220 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.567236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.567977 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.567983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.568194 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.569036 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:26.569069 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:26.571898 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:26.576031 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.576039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.576044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.576455 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.576869 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.576873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.576875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.577089 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.577762 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:26.577766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:26.577930 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 02:34:26.578070 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 02:34:26.578334 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260620 02:34:26.578953 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.563134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.97.61:37613)
0620 02:34:26.563340 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.563345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.563424 (+ 79us) server_negotiation.cc:407] Connection header received
0620 02:34:26.563676 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.563681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.563750 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.563850 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.564874 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.565636 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.567377 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.567814 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.567910 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.572163 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:26.572170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:26.572175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:26.572219 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:26.575840 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.576595 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.576601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.576605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.576754 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.577199 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.577204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.577207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.577591 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:26.577746 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:26.578216 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 02:34:26.578451 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:34:26.580044 23267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:26.581179 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.580695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48378)
0620 02:34:26.580885 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.580977 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:34:26.582109 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.581774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48386)
0620 02:34:26.581934 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.582020 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:34:26.582342 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.633554 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 1
I20260620 02:34:26.634895 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.634268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48396)
0620 02:34:26.634560 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.634680 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:34:26.636097 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.635653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48408)
0620 02:34:26.635852 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.635949 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:34:26.636385 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.666706 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 3
I20260620 02:34:26.667889 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.667407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48414)
0620 02:34:26.667650 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.667732 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:34:26.668969 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.668557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48416)
0620 02:34:26.668784 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.668854 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260620 02:34:26.669214 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.704713 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 2
I20260620 02:34:26.706243 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.705558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48430)
0620 02:34:26.705876 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.706012 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 02:34:26.707356 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.706968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48444)
0620 02:34:26.707138 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.707224 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:34:26.707604 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.812425 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 4
I20260620 02:34:26.813786 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.813174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48454)
0620 02:34:26.813430 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.813570 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:34:26.813879 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 3
I20260620 02:34:26.814990 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.814490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48456)
0620 02:34:26.814676 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.814793 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 02:34:26.815080 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.814690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48464)
0620 02:34:26.814860 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.814939 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 02:34:26.815315 23341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
I20260620 02:34:26.816093 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.815593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48468)
0620 02:34:26.815853 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.815955 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 02:34:26.816327 23267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42885: connect: Connection refused (error 111)
W20260620 02:34:26.838367 23349 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:34:26.838754 23349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:34:26.838888 23349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:34:26.839030 23349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:34:26.846911 23349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:34:26.847014 23349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:34:26.847108 23349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:34:26.847190 23349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:34:26.847307 23349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:34:26.857641 23349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43191,127.0.97.61:37613,127.0.97.60:42885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:34:26.858990 23349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:34:26.860301 23349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:34:26.873091 23355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:34:26.873134 23354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:26.873392 23349 server_base.cc:1034] running on GCE node
W20260620 02:34:26.873088 23357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:34:26.874441 23349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:34:26.875944 23349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:34:26.877174 23349 hybrid_clock.cc:648] HybridClock initialized: now 1781922866877108 us; error 69 us; skew 500 ppm
I20260620 02:34:26.877511 23349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:34:26.880822 23349 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:34:26.882436 23349 webserver.cc:460] Webserver started at http://127.0.97.60:44853/ using document root <none> and password file <none>
I20260620 02:34:26.883183 23349 fs_manager.cc:359] Metadata directory not provided
I20260620 02:34:26.883328 23349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:34:26.883661 23349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:34:26.886803 23349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data/instance:
uuid: "7a4e997a9ce54e759cf309d9941897d2"
format_stamp: "Formatted at 2026-06-20 02:34:26 on dist-test-slave-r706"
server_key: "aac3d07486b461aced8d1ff928a7ce4b"
server_key_iv: "0e3c9f480653d23d0001b8d58999d7de"
server_key_version: "encryptionkey@0"
I20260620 02:34:26.887639 23349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal/instance:
uuid: "7a4e997a9ce54e759cf309d9941897d2"
format_stamp: "Formatted at 2026-06-20 02:34:26 on dist-test-slave-r706"
server_key: "aac3d07486b461aced8d1ff928a7ce4b"
server_key_iv: "0e3c9f480653d23d0001b8d58999d7de"
server_key_version: "encryptionkey@0"
I20260620 02:34:26.892859 23349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:34:26.896474 23364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:34:26.898000 23349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:34:26.898208 23349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal
uuid: "7a4e997a9ce54e759cf309d9941897d2"
format_stamp: "Formatted at 2026-06-20 02:34:26 on dist-test-slave-r706"
server_key: "aac3d07486b461aced8d1ff928a7ce4b"
server_key_iv: "0e3c9f480653d23d0001b8d58999d7de"
server_key_version: "encryptionkey@0"
I20260620 02:34:26.898455 23349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:34:26.926050 23349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:34:26.930667 23349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:34:26.931017 23349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:34:26.950484 23349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42885
I20260620 02:34:26.950490 23415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42885 every 8 connection(s)
I20260620 02:34:26.951921 23349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:34:26.956764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkyewzx/build/asan/bin/kudu as pid 23349
I20260620 02:34:26.957046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkyewzx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922485472743-388-0/minicluster-data/master-2/wal/instance
I20260620 02:34:26.957057 23416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:34:26.957697 388 external_mini_cluster.cc:1428] Setting key 80e9fa5eac9e4b86c7a735d3028de461
I20260620 02:34:26.966342 23416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:34:26.985520 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.959950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.0.97.60:42885)
0620 02:34:26.961284 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.961300 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.961354 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:34:26.963743 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.963767 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.964214 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.964691 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.966256 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.967734 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.969317 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.970195 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.970408 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.974845 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:26.974854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:26.974867 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:26.974909 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:26.979026 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.980419 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.980431 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.980442 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.980815 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.981556 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.981733 (+ 177us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.981745 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.982788 (+ 1043us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:26.983066 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:26.983741 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 02:34:26.984321 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":717,"threads_started":1}
I20260620 02:34:26.987095 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.969091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.97.62:43191)
0620 02:34:26.969648 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.969656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.970492 (+ 836us) server_negotiation.cc:407] Connection header received
0620 02:34:26.970553 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.970556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.970637 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.970769 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.971942 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.972636 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.974598 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.975030 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.975196 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.979353 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:26.979361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:26.979366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:26.979441 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:26.983913 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.984780 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.984785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.984787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.984927 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:26.985398 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:26.985404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:26.985406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:26.985789 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:26.985953 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:26.986457 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 02:34:26.986681 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":316,"threads_started":1}
I20260620 02:34:26.987118 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.969091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43191 (local address 127.0.0.1:53918)
0620 02:34:26.970139 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.970181 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:26.970353 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:26.970839 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:26.970846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:26.971486 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:26.971760 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.971769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.972807 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.972813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.974303 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.974319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.975248 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.975255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.975467 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.976097 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:26.976115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:26.979028 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:26.984112 (+ 5084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.984122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.984128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.984610 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.985096 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:26.985102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:26.985106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:26.985259 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:26.985945 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:26.985953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:26.986133 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 02:34:26.986324 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 02:34:26.986663 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":625,"threads_started":1}
I20260620 02:34:26.988627 23416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:26.993599 23267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 4
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 02:34:26 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:34:27.007314 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.989469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37613 (local address 127.0.0.1:52718)
0620 02:34:26.989759 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.989778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:26.989891 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:26.990228 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:26.990233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:26.990737 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:26.990954 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.990968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.992561 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.992568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.994194 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.994212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.995614 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.995621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.995845 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.996560 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:26.996587 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:26.999503 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:27.004059 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.004065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.004073 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.004471 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.004932 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.004937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.004940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.005184 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.006012 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:27.006018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:27.006199 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 02:34:27.006402 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 02:34:27.006938 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260620 02:34:27.007439 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.989548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.0.97.61:37613)
0620 02:34:26.989729 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.989734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.989912 (+ 178us) server_negotiation.cc:407] Connection header received
0620 02:34:26.990012 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.990015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.990079 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.990167 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.991230 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.992340 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.994389 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.995018 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.995565 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:26.999734 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:26.999740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:26.999750 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:26.999789 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.003907 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.004620 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.004626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.004629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.004777 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.005329 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.005336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.005339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.005732 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.005885 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.006826 (+ 941us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.007083 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:34:27.008632 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.989046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.0.97.62:43191)
0620 02:34:26.989229 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.989234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.989248 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:34:26.989329 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.989334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.989431 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.989538 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.991229 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.992338 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.994723 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.995652 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.995790 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.001275 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.001280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.001283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.001314 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.005177 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.006025 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.006031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.006036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.006171 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.006586 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.006591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.006595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.006958 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.007228 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.008230 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.008412 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:34:27.010099 23416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:34:27.013135 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.994324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48490)
0620 02:34:26.994604 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:26.994624 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:26.994745 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:26.995517 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:26.995523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:26.996125 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:26.996347 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.996362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.997937 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.997944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:26.998972 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:26.998984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.999994 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.000002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:27.000198 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.000923 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:27.000965 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:27.004726 (+ 3761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:27.009622 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.009626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.009630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.010179 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.010842 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.010850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.010856 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.011054 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.011912 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:27.011918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:27.012105 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 02:34:27.012294 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 02:34:27.012680 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 02:34:27.013422 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:26.994400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.0.97.60:42885)
0620 02:34:26.994613 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:26.994622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:26.994680 (+ 58us) server_negotiation.cc:407] Connection header received
0620 02:34:26.994876 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:26.994882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:26.994972 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:26.995082 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:26.996536 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.997765 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.999127 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:26.999767 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:26.999906 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.004946 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.004952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.004957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.005022 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.009445 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.010372 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.010377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.010381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.010658 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.011198 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.011203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.011207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.011692 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.011953 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.012529 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.012874 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:34:27.020509 23267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963: Bootstrap starting.
Jun 20 02:34:27 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:34:27 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:34:27.025595 23267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:27.027179 23267 log.cc:824] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:27.030987 23267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963: No bootstrap required, opened a new log
I20260620 02:34:27.031433 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.010718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.0.97.61:37613)
0620 02:34:27.010950 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.010955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.011342 (+ 387us) server_negotiation.cc:407] Connection header received
0620 02:34:27.011416 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.011422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.011501 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.011637 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.013517 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.014673 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.016527 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.017207 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.017331 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.023707 (+ 6376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.023714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.023721 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.023765 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.028182 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.029179 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.029186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.029193 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.029350 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.029852 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.029857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.029860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.030239 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.030534 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.030852 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.031099 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:34:27.032619 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.011004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48502)
0620 02:34:27.011199 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:27.011217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:27.011318 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:27.014331 (+ 3013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:27.014339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:27.015049 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:27.015300 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:27.015315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.017243 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.017251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:27.019255 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:27.019275 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.020892 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.020901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:27.021294 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.022014 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:27.022047 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:27.025760 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:27.029286 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.029292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.029296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.029857 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.030257 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.030262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.030265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.030410 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.031212 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:27.031217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:27.031373 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 02:34:27.031479 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:34:27.032175 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:34:27.032763 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.011303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.97.60:42885)
0620 02:34:27.013807 (+ 2504us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.013816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.013838 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:34:27.013898 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.013905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.014016 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.014139 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.015484 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.017040 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.019454 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.020447 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.020591 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.026039 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.026045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.026049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.026087 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.029151 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.029991 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.029998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.030002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.030135 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.030543 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.030547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.030549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.030943 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.031773 (+ 830us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.031900 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.032388 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":195,"server-negotiator.queue_time_us":2348,"thread_start_us":14745,"threads_started":1}
I20260620 02:34:27.038762 23416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: Bootstrap starting.
I20260620 02:34:27.039260 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:34:27.041685 23242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7e8a60125c004ba797ef60f9be570963"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926
I20260620 02:34:27.043167 23416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:27.042965 23267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.043396 23267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8a60125c004ba797ef60f9be570963, State: Initialized, Role: FOLLOWER
I20260620 02:34:27.043972 23267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.044639 23416 log.cc:824] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:27.045087 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.045557 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.046418 23267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:27.046567 23242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:34:27.046785 23242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:27.049054 23416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: No bootstrap required, opened a new log
I20260620 02:34:27.050426 23242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.053329 23242 leader_election.cc:288] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [CANDIDATE]: Term 1 election: Requested vote from peers aff52d32fbac4223acfde1757bcc3d70 (127.0.97.61:37613), 7a4e997a9ce54e759cf309d9941897d2 (127.0.97.60:42885)
I20260620 02:34:27.059027 23436 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:34:27.065969 23416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.066570 23416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4e997a9ce54e759cf309d9941897d2, State: Initialized, Role: FOLLOWER
I20260620 02:34:27.067382 23416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
W20260620 02:34:27.068665 23221 tablet.cc:2369] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:34:27.066461 23200 master_runner.cc:186] Error getting master registration for 127.0.97.61:37613: OK, instance_id { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" instance_seqno: 1781922866378658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:34:27.078649 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.080464 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.081489 23200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:34:27.084637 23416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:27.084741 23391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e8a60125c004ba797ef60f9be570963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a4e997a9ce54e759cf309d9941897d2"
I20260620 02:34:27.085464 23391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:27.091881 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.062294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.97.62:43191)
0620 02:34:27.062474 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.062480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.062500 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:34:27.062678 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.062683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.062761 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.062841 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.066622 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.067976 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.070166 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.070909 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.071026 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.076909 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.076916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.076922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.076976 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.081551 (+ 4575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.083597 (+ 2046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.083609 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.083614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.084022 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.089182 (+ 5160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.089190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.089194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.089751 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.090169 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.090961 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.091307 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:34:27.092988 23391 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8a60125c004ba797ef60f9be570963 in term 1.
I20260620 02:34:27.093866 23216 leader_election.cc:302] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a4e997a9ce54e759cf309d9941897d2, 7e8a60125c004ba797ef60f9be570963; no voters:
I20260620 02:34:27.094503 23424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:34:27.097878 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.062783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.0.97.61:37613)
0620 02:34:27.063000 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.063007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.070361 (+ 7354us) server_negotiation.cc:407] Connection header received
0620 02:34:27.070655 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.070666 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.070817 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.070943 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.073619 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.075234 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.077471 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.079624 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.079906 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.087585 (+ 7679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.087595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.087599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.087668 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.091792 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.093715 (+ 1923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.093726 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.093733 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.093998 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.094576 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.094584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.094590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.095128 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.095470 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.096928 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.097333 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:34:27.106467 23424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [term 1 LEADER]: Becoming Leader. State: Replica: 7e8a60125c004ba797ef60f9be570963, State: Running, Role: LEADER
I20260620 02:34:27.106973 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.079663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.0.97.60:42885)
0620 02:34:27.079914 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.079920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.083304 (+ 3384us) server_negotiation.cc:407] Connection header received
0620 02:34:27.083503 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.083509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.083640 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.083763 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.085840 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.087119 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.089565 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.090397 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.090550 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.095102 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.095113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.095121 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.095179 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.100445 (+ 5266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.101637 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.101646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.101652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.101882 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.102567 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.102577 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.102583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.103810 (+ 1227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.104712 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.105169 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.105836 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:34:27.107084 23424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.111668 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e8a60125c004ba797ef60f9be570963. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.112528 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:27.113999 23455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:34:27.114526 23341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } attempt: 5
I20260620 02:34:27.127269 23455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 02:34:27 dist-test-slave-r706 krb5kdc[23187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 02:34:27.127117 23458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 02:34:27.151230 23391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [term 1 FOLLOWER]: Refusing update from remote peer 7e8a60125c004ba797ef60f9be570963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:34:27.152621 23446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:34:27.156747 23458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:34:27.158381 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.115217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.0.97.60:42885)
0620 02:34:27.115389 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.115395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.115686 (+ 291us) server_negotiation.cc:407] Connection header received
0620 02:34:27.115753 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.115760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.115837 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.115929 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.117110 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.118399 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.124124 (+ 5725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.124874 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.125173 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.131483 (+ 6310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:34:27.131490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:34:27.131495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:34:27.131569 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:34:27.136046 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.137734 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.137756 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.137761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.150117 (+ 12356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:34:27.155502 (+ 5385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:34:27.155511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:34:27.155516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:34:27.155981 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:34:27.156122 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:27.157663 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0620 02:34:27.157961 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:34:27.167451 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:27.115165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42885 (local address 127.0.0.1:48524)
0620 02:34:27.115434 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 02:34:27.115453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:34:27.115578 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:34:27.115994 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:34:27.116001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:34:27.116607 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:34:27.116828 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:27.116842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.121691 (+ 4849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.121701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:27.123832 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:34:27.123854 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:34:27.125435 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:34:27.125442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:34:27.125831 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:27.126896 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:34:27.126952 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:34:27.131169 (+ 4217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:34:27.136434 (+ 5265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.136440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.136444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.137507 (+ 1063us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.154064 (+ 16557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:34:27.154074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:34:27.154078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:34:27.155082 (+ 1004us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:34:27.156410 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:34:27.156415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:34:27.157193 (+ 778us) client_negotiation.cc:769] Sending connection context
0620 02:34:27.157504 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 02:34:27.157886 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 02:34:27.174052 23341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: Bootstrap starting.
I20260620 02:34:27.178510 23341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: Neither blocks nor log segments found. Creating new log.
I20260620 02:34:27.180151 23341 log.cc:824] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: Log is configured to *not* fsync() on all Append() calls
I20260620 02:34:27.131395 23349 master_runner.cc:186] Error getting master registration for 127.0.97.61:37613: OK, instance_id { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" instance_seqno: 1781922866378658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260620 02:34:27.183303 23370 tablet.cc:2369] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:34:27.183964 23349 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:34:27.188920 23341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: No bootstrap required, opened a new log
I20260620 02:34:27.197019 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e8a60125c004ba797ef60f9be570963. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.197386 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.198302 23461 mvcc.cc:205] Tried to move back new op lower bound from 7298756063810170880 to 7298756063676682240. Current Snapshot: MvccSnapshot[applied={T|T < 7298756063810170880}]
I20260620 02:34:27.200182 23462 mvcc.cc:205] Tried to move back new op lower bound from 7298756063810170880 to 7298756063676682240. Current Snapshot: MvccSnapshot[applied={T|T < 7298756063810170880}]
I20260620 02:34:27.200671 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.200977 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:27.205418 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.205699 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.212128 23455 catalog_manager.cc:1347] Generated new cluster ID: 45feb64815f14092b2d41744373cdd90
I20260620 02:34:27.212272 23455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:34:27.213927 23341 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.214360 23341 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff52d32fbac4223acfde1757bcc3d70, State: Initialized, Role: FOLLOWER
I20260620 02:34:27.214938 23341 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } }
I20260620 02:34:27.222399 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.225492 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.225292 23341 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:34:27.225708 23315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:34:27.230792 23315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [term 1 FOLLOWER]: Refusing update from remote peer 7e8a60125c004ba797ef60f9be570963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:34:27.232326 23424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:34:27.233309 23316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e8a60125c004ba797ef60f9be570963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aff52d32fbac4223acfde1757bcc3d70"
I20260620 02:34:27.244675 23316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8a60125c004ba797ef60f9be570963 in term 1.
I20260620 02:34:27.294243 23455 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:34:27.296411 23455 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:34:27.298777 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e8a60125c004ba797ef60f9be570963. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.299237 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.299772 23475 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.300304 23475 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:34:27.301972 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e8a60125c004ba797ef60f9be570963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8a60125c004ba797ef60f9be570963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43191 } } peers { permanent_uuid: "aff52d32fbac4223acfde1757bcc3d70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37613 } } peers { permanent_uuid: "7a4e997a9ce54e759cf309d9941897d2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42885 } } }
I20260620 02:34:27.302286 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963 [sys.catalog]: This master's current role is: LEADER
I20260620 02:34:27.323216 23455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e8a60125c004ba797ef60f9be570963: Generated new TSK 0
I20260620 02:34:27.324508 23455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:34:27.362483 23483 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 02:34:27.443825 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.438702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.97.61:37613)
0620 02:34:27.438980 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.438986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.441695 (+ 2709us) server_negotiation.cc:407] Connection header received
0620 02:34:27.442038 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.442046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.442140 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.442221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.443563 (+ 1342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52748: BlockingRecv error: recv got EOF from 127.0.0.1:52748 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 02:34:27.454481 23420 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.438232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.97.62:43191)
0620 02:34:27.438519 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.438528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.451597 (+ 13069us) server_negotiation.cc:407] Connection header received
0620 02:34:27.452221 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.452231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.452355 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.452470 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.454129 (+ 1659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53942: BlockingRecv error: recv got EOF from 127.0.0.1:53942 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:34:27.454849 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.437335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.97.60:42885)
0620 02:34:27.438017 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.438027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.445535 (+ 7508us) server_negotiation.cc:407] Connection header received
0620 02:34:27.453360 (+ 7825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.453367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.453487 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.453576 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.454571 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48536: BlockingRecv error: recv got EOF from 127.0.0.1:48536 (error 108)
Metrics: {"server-negotiator.queue_time_us":508}
W20260620 02:34:27.462435 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.460040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.97.61:37613)
0620 02:34:27.460297 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.460306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.460802 (+ 496us) server_negotiation.cc:407] Connection header received
0620 02:34:27.461109 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.461114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.461235 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.461367 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.462144 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52762: BlockingRecv error: recv got EOF from 127.0.0.1:52762 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 02:34:27.468596 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.466338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.0.97.61:37613)
0620 02:34:27.466564 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.466572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.467087 (+ 515us) server_negotiation.cc:407] Connection header received
0620 02:34:27.467310 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.467320 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.467393 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.467517 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.468256 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52766: BlockingRecv error: recv got EOF from 127.0.0.1:52766 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 02:34:27.476329 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.474095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.0.97.61:37613)
0620 02:34:27.474333 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.474342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.474854 (+ 512us) server_negotiation.cc:407] Connection header received
0620 02:34:27.475112 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.475123 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.475207 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.475328 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.476072 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52780: BlockingRecv error: recv got EOF from 127.0.0.1:52780 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:34:27.487152 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.484798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.0.97.61:37613)
0620 02:34:27.485100 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.485110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.485654 (+ 544us) server_negotiation.cc:407] Connection header received
0620 02:34:27.485898 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.485904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.485996 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.486119 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.486886 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52784: BlockingRecv error: recv got EOF from 127.0.0.1:52784 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 02:34:27.501878 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.499505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.0.97.61:37613)
0620 02:34:27.499748 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.499756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.500346 (+ 590us) server_negotiation.cc:407] Connection header received
0620 02:34:27.500588 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.500596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.500681 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.500805 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.501633 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52788: BlockingRecv error: recv got EOF from 127.0.0.1:52788 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:34:27.524113 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.521771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.0.97.61:37613)
0620 02:34:27.522015 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.522024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.522591 (+ 567us) server_negotiation.cc:407] Connection header received
0620 02:34:27.522845 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.522851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.522961 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.523083 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.523859 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52802: BlockingRecv error: recv got EOF from 127.0.0.1:52802 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 02:34:27.562605 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.559930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.0.97.61:37613)
0620 02:34:27.560192 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.560201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.560763 (+ 562us) server_negotiation.cc:407] Connection header received
0620 02:34:27.561027 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.561035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.561128 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.561248 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.562304 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52818: BlockingRecv error: recv got EOF from 127.0.0.1:52818 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 02:34:27.641536 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.639022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.0.97.61:37613)
0620 02:34:27.639268 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.639274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.639865 (+ 591us) server_negotiation.cc:407] Connection header received
0620 02:34:27.640088 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.640095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.640197 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.640317 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.641291 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52822: BlockingRecv error: recv got EOF from 127.0.0.1:52822 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 02:34:27.777125 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:27.775078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.97.61:37613)
0620 02:34:27.775352 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:27.775360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:27.775596 (+ 236us) server_negotiation.cc:407] Connection header received
0620 02:34:27.775841 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:27.775845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:27.775920 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:27.776037 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:27.776809 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52832: BlockingRecv error: recv got EOF from 127.0.0.1:52832 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 02:34:28.040199 23346 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:34:28.038312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.0.97.61:37613)
0620 02:34:28.038495 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:28.038500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:28.038837 (+ 337us) server_negotiation.cc:407] Connection header received
0620 02:34:28.039038 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:28.039043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:28.039128 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:28.039228 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:34:28.039932 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52848: BlockingRecv error: recv got EOF from 127.0.0.1:52848 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:34:28.159382 23458 catalog_manager.cc:1259] Loaded cluster ID: 45feb64815f14092b2d41744373cdd90
I20260620 02:34:28.159518 23458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: loading cluster ID for follower catalog manager: success
I20260620 02:34:28.162206 23458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: acquiring CA information for follower catalog manager: success
I20260620 02:34:28.164155 23458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a4e997a9ce54e759cf309d9941897d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:34:28.367413 23483 catalog_manager.cc:1259] Loaded cluster ID: 45feb64815f14092b2d41744373cdd90
I20260620 02:34:28.367678 23483 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: loading cluster ID for follower catalog manager: success
I20260620 02:34:28.371393 23483 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: acquiring CA information for follower catalog manager: success
I20260620 02:34:28.373631 23483 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aff52d32fbac4223acfde1757bcc3d70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:34:28.564235 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:28.556748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.97.61:37613)
0620 02:34:28.557853 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:28.557858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:28.557877 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:34:28.557942 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:28.557946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:28.558086 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:28.558191 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:28.558768 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.559580 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.561714 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.562191 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.562301 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:28.562918 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:28.563478 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:28.563600 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:28.563744 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 02:34:28.563822 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":843,"threads_started":1}
I20260620 02:34:28.577534 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:28.570994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.97.60:42885)
0620 02:34:28.571553 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:28.571558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:28.571576 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:34:28.571667 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:28.571671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:28.571837 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:28.571983 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:28.572496 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.573324 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.575234 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.575623 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.575718 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:28.576283 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:28.576854 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:28.576974 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:28.577113 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 02:34:28.577186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":327,"threads_started":1}
I20260620 02:34:28.603086 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:28.584825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.97.60:42885)
0620 02:34:28.592588 (+ 7763us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:28.592597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:28.595928 (+ 3331us) server_negotiation.cc:407] Connection header received
0620 02:34:28.596184 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:28.596189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:28.596295 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:28.596389 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:28.597194 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.598399 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.600725 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.601410 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.601509 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:28.602153 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:28.602321 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:28.602387 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:28.602651 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 02:34:28.602749 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7556}
I20260620 02:34:28.603565 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:28.585094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.97.62:43191)
0620 02:34:28.595509 (+ 10415us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:28.595517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:28.595545 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:34:28.595696 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:28.595705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:28.595961 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:28.596175 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:28.596754 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.597998 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.600363 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.601047 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.601173 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:28.601954 (+ 781us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:28.602513 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:28.602616 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:28.603090 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 02:34:28.603185 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10263,"thread_start_us":10143,"threads_started":1}
I20260620 02:34:28.603811 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:34:28.590923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.0.97.61:37613)
0620 02:34:28.591213 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 02:34:28.591223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:34:28.596076 (+ 4853us) server_negotiation.cc:407] Connection header received
0620 02:34:28.596869 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:34:28.596876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:34:28.596983 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:34:28.597089 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:34:28.597910 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.599033 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.601282 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:34:28.601960 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:34:28.602092 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:34:28.602790 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:34:28.602923 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:34:28.603008 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:34:28.603380 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 02:34:28.603482 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:34:28.610076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 23200
I20260620 02:34:28.611166 23262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:28.767019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 23200
W20260620 02:34:28.789969 23366 connection.cc:535] server connection from 127.0.0.1:48490 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:34:28.790428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 23274
I20260620 02:34:28.791576 23336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:28.901664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 23274
I20260620 02:34:28.924530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkyewzx/build/asan/bin/kudu pid 23349
I20260620 02:34:28.925745 23411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:34:29.037206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkyewzx/build/asan/bin/kudu with pid 23349
2026-06-20T02:34:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5124 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10089 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (383487 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST